Someone did something on Friday afternoon, and now every DTA on the network is prompting for Bitlocker drive recovery.
Remember - we have about 8,000 active users.
At about 9:30 Monday morning I saw the largest number of calls waiting, and the longest wait time, that I have ever seen. And that was 27 calls waiting, over 11 minutes. And right now, 13 calls with 7 minutes wait.
What fun.
What does "DTA" expand to? I'm not getting any success on DuckDuckGo when pairing it with "Windows", "network," or "Bitlocker."
When someone calls, we have to provide them with a 48-digit drive recovery key. There's a lot of people in this room just chanting numbers.
Looking at this through the lens of a long-time Linux user, I wonder how this problem and the solution would have played out in the Linux world.
First we assume that an error of the same magnitude happened: someone did something to a server and borked access to a critical service used by 8,000 users. We further assume a fix can be put together over the weekend without having to go through a massive bureaucracy for implementing it, which simply may not be possible in a government organization with 8,000 users.
Given that you mention giving a 48 digit code over the phone I'm assuming users can start the login process on their workstation, although they may be immediately be given a dialogue box asking for the recovery key.
Back to Linux. Under Linux the
ssh service starts as soon as networking is available, meaning technical people with the correct password (or better, correct ssh private key) can connect to the system.
Provided you have technical I/T people working Saturday and Sunday, they could put together a script to identify the machine, retrieve the unlock key from whatever the organization uses for Active Directory, and apply the key to unlock the volume. Push out the script to all 8,000 systems over the weekend and on Monday morning people log in and don't see any problem.
All right, suppose the majority of affected computers are inaccessible because they were down for the weekend. The technical people could add the unlock script to the base system profile on the central directory server, meaning it would be run when the computer connects, By the time the user has entered their credentials the script has finished its work and the user is able to log in and start working.
Another possible solution, in the event that a script-based unlock is not possible. (Which I rather doubt would be the case in Linux. Pretty much every Linux service can be managed from the command line. Any GUI front end interacts with the Linux service using the same system calls the command line utility uses, and some GUIs actually do their work by issuing command line instructions.) Assuming there are technical I/T people working on the weekend and you're running a capable PBX like Asterisk, someone could put together a script the retrieve the access key and send it to the affected system. Make sure the script is in place on Monday morning and add a new menu option to the top level help desk IVR (Interactive Voice Response) menu when the calls start coming in.
The tough part of designing and writing that script would be verifying callers and the computer they regularly use to ensure someone isn't using the current crisis to unlock someone else's system. One could also push that part of the recovery process to the Help Desk people, and once they've validated the identity can transfer the call to the script that performs the unlock, freeing up help desk staff to go to the next call.
Arthwollipot, I realise you're a Help Desk person and not on the technical team, but it sounds like your technical people rely too much on the Help Desk to dig them out of their blunders instead of searching for and implementing a scripted solution.
For people with more experience managing large AD setups: are scripted solutions like the ones I described available in this environment? Could they be implemented quickly enough that the Help Desk is not inundated with calls, especially given the two day lead time?
