Dear Users... (A thread for Sysadmin, Technical Support, and Help Desk people)

[Wudang]

Dear Users.

No IT Professional has ever, in the recent history of this or any possible universe, ever told users on a normal, business office Windows Domain/Server/Workstation network to "shut down at the end of the day and leave their computers off until the next morning."

Don't whine at me when you catch an update in the middle of the day and Windows reboots. Especially on a Monday if you're computer was turned off ever since you cut out for lunch early on Friday.

HSBC offices were ordered to put up posters telling staff to do exactly that. There was one outside each machine room in the data centre I worked in. Of course that may not have been ordered by an "IT professional" or even a competent one.

 

[Blue Mountain]

Sorry, it's Desktop Anywhere, which is the Microsoft protocol to allow a device to connect to a secure network over any unsecure Internet connection. In this organisation it's common to refer to any device - usually a laptop or a Surface Pro - running this protocol as a DTA. So in this case, DTA basically means laptop. Sorry for thoughtlessly using a local colloquialism.

Thanks for the explanation.

In some cases it is possible, certainly. But there are also cases where the damage is done in an instant and the only thing we can do is clean up after it. Bitlocker is a required system (if any reader doesn't know what that is, it encrypts the hard drive of a device so that if the device is ever lost or stolen, the bad guys can't get illegal access to its contents), and it requires the recovery key to proceed. We are the only ones who can provide that key, and that is by design. By forcing the user to come back to us for verification, we can ensure that only legitimate users can access the device.

Sounds like that part was working as expected.

Part of my musings for an emergency fix (on Linux) was to write an Asterisk script that would pull the recovery key from somewhere and read off the numbers. It would be tricky, but I'm sure there's some way the Help Desk could be given an emergency interface where once they had verified the caller, they could copy the recovery key from AD, paste it to a dialogue box, then transfer the call to a temporary extension implementing the readout script. That would let the Help Desk person to go on to the next call. I'm not sure if all 8,000 users were affected, but if they were such a quick fix would certainly help with processing the calls.

Of course this assumes your I/T people have access to the workings of your phone system. That's not guaranteed.

 

[arthwollipot]

Training in my new position is proceeding apace. I can now assign a RAS token without referring to the operating procedures. What I'm doing now is working through a number of requests so that I can determine the best order of operations to maximise the efficiency of my clipboard.

 

[novaphile]

Jumping in to say that we're also required to power down for extended absences (i.e. overnight) and all of our computers have wake on LAN enabled.

The only time I've had to leave mine on overnight was when uploading a log file to a vendor on the other side of the world. (Yes, it really was that big)

 

[Blue Mountain]

Jumping in to say that we're also required to power down for extended absences (i.e. overnight) and all of our computers have wake on LAN enabled.

That policy makes for an interesting intersection between the dollar cost of running the computers and the time taken to start up the system. Now the time factor would be greatly reduced if you could suspend or hibernate the OS and desktop environment as opposed to shutting the whole thing down. However, if every morning when coming into work you had to start your computer, log in, start up email, word processor, web browser, text editor, an IDE, team productivity app (e.g. Slack), perhaps a softphone app, and maybe a Help Desk ticketing app (if i's not browser based,) the time wasted every day doing that would quickly exceed the minor cost in electricity used to run the computers overnight.

I recall a case a while ago from the States where employees in a call centre started at 9:00 AM. But the expectation was they were ready to take calls at 9:00 AM, which meant they had to arrive at work five to ten minutes earlier to get ready. They sued their employer to get paid for those additional the ten minutes and won.

 

[novaphile]

That policy makes for an interesting intersection between the dollar cost of running the computers and the time taken to start up the system. Now the time factor would be greatly reduced if you could suspend or hibernate the OS and desktop environment as opposed to shutting the whole thing down. However, if every morning when coming into work you had to start your computer, log in, start up email, word processor, web browser, text editor, an IDE, team productivity app (e.g. Slack), perhaps a softphone app, and maybe a Help Desk ticketing app (if i's not browser based,) the time wasted every day doing that would quickly exceed the minor cost in electricity used to run the computers overnight.

I recall a case a while ago from the States where employees in a call centre started at 9:00 AM. But the expectation was they were ready to take calls at 9:00 AM, which meant they had to arrive at work five to ten minutes earlier to get ready. They sued their employer to get paid for those additional the ten minutes and won.

Interesting...

You may be forgetting the cost of the air-conditioning to keep the building cool in response to all those computers sitting there humming away.

In terms of lost time starting up the computers... if it takes you ten minutes to power up your computer and get ready for work, there's something wrong with you and your computer.

When I get to work, I hit the power switch on my computer.
I take my phone out of my bag, and put my bag into the cupboard at my desk then sit down.
The computer is up and running and everything is ready to go.

If yours isn't working that well, you could always use the task scheduler.

All this will be moot for us soon, there is some consideration to moving everything to the computer room, and only run the minimum required on the local computer to connect to the virtual machine...

 

[arthwollipot]

I'm going mad. We have a dummy ID that we use to log calls for people who are not actually on our network (and we have quite a few of those because reasons). It's ext123. Yesterday I swear I tried that and it didn't work, and I had to use ext321.

Obviously that wasn't the case, but damn this is some Mandela Effect stuff going on here.

 

[arthwollipot]

[imgw=300]https://i.imgur.com/a4uo7WF.gif[/imgw]

Okay. Wanna hear about the latest fustercluck? This is a good one.

This week we at the Service Desk received notification that the department is replacing Adobe Acrobat Professional with a replacement product called Kofax PowerPDF. It's supposed to be a fully-functional replacement. The change is being made because as you know, Adobe software is hideously expensive, more so in Australia because they can. That's literally the reason Adobe gave a Parliamentary Committee for charging so much more for their software in Australia than they do in other markets.

It was supposed to be installed to all computers that have Acrobat Pro at the beginning of next week, and Acrobat Pro removed from those computers the week after that.

All 1200 users of Adobe Acrobat Pro have had it removed this morning, and no replacement is available.

But that's not even the best part. You want to know the best part? Of course you do. None of the users were told anything about this change. The media team is furious. Not only because they literally can't do their work this morning, but also because an arbitrary decision has been made to move away from an industry standard product with no warning and no consultation.

There are insufficient facepalms and headdesks to express how I'm feeling right now.

 

[Blue Mountain]

Interesting...

You may be forgetting the cost of the air-conditioning to keep the building cool in response to all those computers sitting there humming away.

In Manitoba in the winter that isn't too much of a concern In Texas in the summer it probably would be.

In terms of lost time starting up the computers... if it takes you ten minutes to power up your computer and get ready for work, there's something wrong with you and your computer.

Old age—both the computer and the user.

When I get to work, I hit the power switch on my computer.
I take my phone out of my bag, and put my bag into the cupboard at my desk then sit down.
The computer is up and running and everything is ready to go.

Nice. Does the computer (or your roaming profile) start up the needed programs? Before or after to log into the desktop?

If yours isn't working that well, you could always use the task scheduler.

I've had very poor success with the Windows task scheduler. My experience is you set it up, it works for about the first three or four times it's supposed to, and then it stops working for no apparent reason (but usually security related.)

All this will be moot for us soon, there is some consideration to moving everything to the computer room, and only run the minimum required on the local computer to connect to the virtual machine...

Yeah. The 1970s called; it wants its mainframe back.

 

[novaphile]

Can't disagree with the last one.

Glad I'm retiring soon. Sick of watching all the old mistakes getting repeated again, and again, and again.

Defn: The Cloud - it didn't work on our mainframe, let's put it on somebody else's mainframe.

We have a "managed private cloud" everything now takes twenty times longer, at four times the cost.

 

[arthwollipot]

We've moved just about everything to a cloud service, and it works pretty well. There are a few annoying bugs, but mostly everything ticks along just fine.

Mostly.

 

[novaphile]

I'm half convinced it's the roaming profiles that are causing all the profile corruption at work.

Usual problem, the profile gets borked, submit a request to desktop and wait for several weeks until someone fixes it.

I hate the new Muppets-in-command.

 

[arthwollipot]

I'm half convinced it's the roaming profiles that are causing all the profile corruption at work.

Usual problem, the profile gets borked, submit a request to desktop and wait for several weeks until someone fixes it.

I hate the new Muppets-in-command.

Several weeks! Don't you have SLAs?

 

[novaphile]

Several weeks! Don't you have SLAs?

We do but they appear to be ignored.

I may have mentioned that the last five years have been clusterfucked because of the new rule that all IT personnel have to use locked-down computers so that we can't fix in one second a problem that takes helpdesk/desktop several weeks to get around to (and then fix in one second).

 

[novaphile]

Of course, the people building and maintaining the giant system that everyone outside of IT has to use 24x7, aren't considered to be core business.

But at the same time, have to use standard build locked down desktops...

By the way, the certificate *********** continues, this time they took SVN out for a week.

 

[Gord_in_Toronto]

We've moved just about everything to a cloud service, and it works pretty well. There are a few annoying bugs, but mostly everything ticks along just fine.

Mostly.

And then, one day, someone nefarious, for nefarious reasons will bring the Internet down really hard. If it cannot be brought up with a matter of minutes, everything stops.

 

[arthwollipot]

And then, one day, someone nefarious, for nefarious reasons will bring the Internet down really hard. If it cannot be brought up with a matter of minutes, everything stops.

Wasn't the internet purposefully designed to be resistant to such attacks?

 

[Blue Mountain]

Wasn't the internet purposefully designed to be resistant to such attacks?

If I remember correctly, one of the DARPA design goals was to "route around" damaged portions. So if the network determined a particular route wasn't available, there might be others that could be used. How automatic that is I don't know.

That works when getting packets from point A to B via C, D, and E, or perhaps C, F, G, and E. But if the only fibre-optic cable serving your area gets cut, there's no way to get from A to anywhere. (That happened recently in the Yukon in Canada.)

The above applies only to routing packets. If where you want to surf to has a power outage with no fail-over site to serve it from, there's no content. If by mistake or malice someone publishes a bad BGP (Border Gateway Protocol) update, some high level routers could go offline or get overwhelmed (that's happened.) A site forgetting to renew a TLS certificate is still "up" (you can reach it on the internet) but your browser may refuse to access the content. (That's happened to Microsoft sites on more than one occasion.)

 

[Faydra]

DR test this past week.

I told people in meetings, don't do x.
I sent emails, don't do x.
I sent a video on how to do y instead to everyone.
X will not work people, please don't do it.


How many people do you think tried to do x?

All of them.

THEN, I see a problem and in the slack channel I say "Hey @user, I see you are trying to do this, it won't work, this is what you need to do to solve it". The very next message in the channel was that user saying "OMG, this isn't working what do I do????".

I love being ignored.

 

[JoeMorgue]

Wasn't the internet purposefully designed to be resistant to such attacks?

Not exactly.

"The Internet" (as nebalous as that term has gotten) in its original concept was just what is known as a Mesh Network.

You have 10 cities networked, each city has a link to every other (or most every other as Mesh Networks are almost never 100% complete) so if one city gets nuked the other 9 can still talk.

There are still chokepoints and single points of failure. And hell mass swaths of the internet get taken down all the time, by governments, by submarine accidentally cutting cables, by bad DNS or routing entries, by sharks, by old ladies stealing copper.