Dear Users... (A thread for Sysadmin, Technical Support, and Help Desk people)

[arthwollipot]

ServiceNow, like most ticketing systems, is full of useful features to streamline workflow.

Unfortunately, like most ticketing systems, the first thing the customer always does is customize it to the point where it's completely unusable by anyone sane.

Every ticketing system I've ever used has had an integrated knowledge base. None of them have been used. All have used custom external knowledge bases. The one I'm using now is Microsoft Service Manager Console. Which is... special.

 

[Faydra]

More super fun software I have had the !pleasure of working with...

Pega (sigh...)
Cyberark (please kill me)
Version One (just one bullet, that's all I ask)
GRC (only thing that will help this is a flamethrower)
Keylight (WTF were they thinking???)

 

[Hellbound]

Service Now is horrendous at best but at least we stopped using release packages.

The Snow Release Package is the 7th level of hell.

Really? I've actually been enjoying it so far. We've streamlined a lot of process, started gathering data in a central location that's meaningful and actually useful, and several other options. It's a lot more than just a ticketing system.

Now, if it's not implemented well...yeah, it can go bad very quickly. The first step is getting a good, clean CMDB, which requires good policies for new systems, retiring systems, and good discovery rules. If your CMDB isn't right everything will go to hell at a speed that leave hand baskets in the dust.

And of course, I'm the platform admin for our ServiceNow instances, so my perspective may be a bit different.

 

[Wudang]

Really? I've actually been enjoying it so far. We've streamlined a lot of process, started gathering data in a central location that's meaningful and actually useful, and several other options. It's a lot more than just a ticketing system.

Now, if it's not implemented well...yeah, it can go bad very quickly. The first step is getting a good, clean CMDB, which requires good policies for new systems, retiring systems, and good discovery rules. If your CMDB isn't right everything will go to hell at a speed that leave hand baskets in the dust.

And of course, I'm the platform admin for our ServiceNow instances, so my perspective may be a bit different.

HSBC uses (used?) Interlink Software's SCM. Their pricing model was a bit off but they changed it after a chat. Allows lot of automated imports of data feeds from BMC, IBM etc products and interfaces with their MOM product so alerts get triaged according to the service info extracted from SCM.

 

[Hellbound]

HSBC uses (used?) Interlink Software's SCM. Their pricing model was a bit off but they changed it after a chat. Allows lot of automated imports of data feeds from BMC, IBM etc products and interfaces with their MOM product so alerts get triaged according to the service info extracted from SCM.

We've done most of ours in-house. It was awful when we started, as this was going to replace our current ticketing system and the contract was coming up, and no one wanted to spend money for another year of a system that we were getting rid of. So we kicked it off with no real CMDB and played by ear for two years. That was awful.

We ended up hiring some specialists to come in and help us clean up our CMDB and discovery, define processes, and get a lot of it set up to match our company policies. It's made a huge difference.

Policy and procedure, and especially change control, are vitally important as well. WE created a Platform Governance Board that includes me as platform admin, each module owner, and reps from other important areas (like development) to approve requested changes to the platform and so forth. If nothing else, the interchange of information between all our groups is a huge benefit on it's own.

Of course, our Security Team are turning into tin-pot dictators (actively excluding technical experts from decisions they make based on very little technical knowledge, over-riding reasonable plans with nonsense, and showing no consistency in how policy is applied or even what current policy is), so as much as I've enjoyed my job here, I'm thinking it's time to brush up the old resume. I really liked this job, but it's getting to the point that every day is a new frustration dealing with the stupid.

Sorry, got a bit off-topic there. Carry on

 

[Wudang]

Of course, our Security Team are turning into tin-pot dictators (actively excluding technical experts from decisions they make based on very little technical knowledge, over-riding reasonable plans with nonsense, and showing no consistency in how policy is applied or even what current policy is), so as much as I've enjoyed my job here, I'm thinking it's time to brush up the old resume. I really liked this job, but it's getting to the point that every day is a new frustration dealing with the stupid.

Sorry, got a bit off-topic there. Carry on

If I get annoyed enough I'll tell you about trying to get changes made to bank's secondary DB security policy so we could install and manage 3rd party products with dynamic schema. 2 years. Total lack of engagement and no interest in use cases. A change was made but it was compounded stupidity that forced me to get all Heath Robinson on a system that obeyed all their rules and worked but ....(cough) best to say no more. Anyway I raised that a few times when people said "if you don't like it make a change" arranged to the tune of "**** that for a game soldiers".



In my work with a medical infomatics charity for which I am employee 4, the process consists of opening a Slack chat with Sophie "hi, is this a good idea or can you do better?".

 

[Hellbound]

If I get annoyed enough I'll tell you about trying to get changes made to bank's secondary DB security policy so we could install and manage 3rd party products with dynamic schema. 2 years. Total lack of engagement and no interest in use cases. A change was made but it was compounded stupidity that forced me to get all Heath Robinson on a system that obeyed all their rules and worked but ....(cough) best to say no more. Anyway I raised that a few times when people said "if you don't like it make a change" arranged to the tune of "**** that for a game soldiers".



In my work with a medical infomatics charity for which I am employee 4, the process consists of opening a Slack chat with Sophie "hi, is this a good idea or can you do better?".

Heh, yeah. In a large company though, you need processes and policies. With small groups you can work around it a bit easier.

On the other hand, we have ~5k employees and a similar number of computer systems. We have SLAs for some systems that assess penalties if it takes more than 13 seconds for a respond to reach back to the point of origin. We handle health and financial information. We have to have good security. But our people just aren't getting there. The policies seem punitive and pointless.

For example, I was getting approval for the public-facing URL of our new single-sign on server (SAML 2.0). This is a URL that is intended to be given to 3rd parties, which MUST be publicly available to the internet to do what we want it to do, and so on. They disapproved of me calling it "sso.company.com" (like literally every other organization structures it) because "that tells an attacker exactly what it is". They wanted it named after something like a muppet of greek gods or whatever.

For a public URL.

Seriously, if the attacker is so bad that he can't footprint and enumerate the URL inside of 5 minutes, then he isn't a threat to us. If they are going to be at all confused or slowed down because the URL name doesn't say what it is, then they are not a danger.

Heck, I wanted to argue that we need to take the "www" of all our website,s because that just tells an attacker what they are.

Infuriating.

And You know, I used to be the Information Security NCOIC for a military unit in the U.S. Army. I know a bit about IT security. They consistently go for appearance over actuality. Drives me insane (and trust me, for me that's walking distance...driving me is a very short trip ).

 

[zooterkin]

ServiceNow, like most ticketing systems, is full of useful features to streamline workflow.

Unfortunately, like most ticketing systems, the first thing the customer always does is customize it to the point where it's completely unusable by anyone sane.

Usually, in my experience, by customising it to look like the previous ticketing system that it’s replacing, despite the fact that makes it run slowly and loses most of the benefits of the new system.

 

[arthwollipot]

Oh hello, I'm calling on behalf of <another user>. They're clicking links and it's not working.

Okay... I'm going to know a lot more detail than that. What's happening when they click the link?

It doesn't work. They don't see what they need to see.

What do they see instead? Does it generate an error message? Does it redirect to a different page? What happens when they click the link.

It just doesn't work.

I really need to know this. I can't determine what's going wrong without a lot more detail. Can you get them to call us directly?

Okay, hold on.

*5 minutes of hold music*

Okay, it says "you do not have permission to access this page".

Alright, that's a start. Is it a Sharepoint page?

Yes, yes. It's Sharepoint.

Okay. Access to Sharepoint pages is determined by the staff who have authority over the data. That's someone in your section. We don't have the authority over ownership of the data, only over the infrastructure.

Okay. So can you fix the problem?

*facepalm*

 

[Norman Alexander]

Oh hello, I'm calling on behalf of <another user>. They're clicking links and it's not working.

Okay... I'm going to know a lot more detail than that. What's happening when they click the link?

It doesn't work. They don't see what they need to see.

What do they see instead? Does it generate an error message? Does it redirect to a different page? What happens when they click the link.

It just doesn't work.

I really need to know this. I can't determine what's going wrong without a lot more detail. Can you get them to call us directly?

Okay, hold on.

*5 minutes of hold music*

Okay, it says "you do not have permission to access this page".

Alright, that's a start. Is it a Sharepoint page?

Yes, yes. It's Sharepoint.

Okay. Access to Sharepoint pages is determined by the staff who have authority over the data. That's someone in your section. We don't have the authority over ownership of the data, only over the infrastructure.

Okay. So can you fix the problem?

*facepalm*

Answer: Your boss can. Have you asked them?

 

[arthwollipot]

Answer: Your boss can. Have you asked them?

Their boss was the one having the problem.

It was a case of the boss saying "I can't open links. Call the Service Desk for me. I have to go to this meeting."

 

[arthwollipot]

I feel like a bit of a zoo animal right now. The PTBs have enriched my environment by changing all of our resolution categories so I have to actually hunt for the category I need.

 

[Wudang]

I feel like a bit of a zoo animal right now. The PTBs have enriched my environment by changing all of our resolution categories so I have to actually hunt for the category I need.

The people at HSBC who decided close codes were almost exclusively people with no idea how software or hardware worked. I tried explaining orthogonal defect classification and got the old deer in headlights look. Anyway we started assigning root cause as "console harness". We had no idea if "console" was even a noun or a verb but it got the tickets closed.

 

[arthwollipot]

The people at HSBC who decided close codes were almost exclusively people with no idea how software or hardware worked. I tried explaining orthogonal defect classification and got the old deer in headlights look. Anyway we started assigning root cause as "console harness". We had no idea if "console" was even a noun or a verb but it got the tickets closed.

Haha. Ours really haven't changed, but they've been given a category according to the Call Type field, so they're all out of order now. Last time we got a major update to the close categories we were briefed on the change beforehand.

 

[arthwollipot]

"Console harness". I'll have to remember that one.

 

[Wudang]

And You know, I used to be the Information Security NCOIC for a military unit in the U.S. Army. I know a bit about IT security. They consistently go for appearance over actuality. Drives me insane (and trust me, for me that's walking distance...driving me is a very short trip ).

Oh yes. I used to be a pentester with IBM and built security for some large systems for IBM UK's business side. I recall one audit by corporate IT Security. All accountants. All telling us that the "Dolan document" (named after its eponymous author) meant we must do x, we had no choice but to follow the commandents from on high. This was relayed by my manager who expressed frustration that they wouldn't even let him read it. I produced my copy and showed how it was actually good. Lots of guiding principles and no commandments. So we phoned Dolan and explained what we were doing and he said that sounded good and he'd update his doc. The look on the auditors faces when we told them they needed to consult the new version when it came out was priceless.

 

[arthwollipot]

I'm having a bit of trouble right now with things being changed and us not being told that they've been changed. Today I've just had to find out for myself that XenMobile is now called Secure Mail, and all of our access forms and knowledge base articles have been changed to reflect that. So I was just in the position of telling someone to submit a XenMobile Access Request form, only to have them tell me there's no such form.

That's a minor issue, and I was fortunately able to resolve it myself, but it's part of a pattern.

 

[Dancing David]

Their boss was the one having the problem.

It was a case of the boss saying "I can't open links. Call the Service Desk for me. I have to go to this meeting."

Oy vey!

 

[Faydra]

Me: The Change Request # you gave me is invalid, it's been closed for over a year.

User: Thanks for the heads up. I’m not actually sure what the change ticket is used for as I was told to just take a previously used ticket and increment it. Can I just keep using a ticket I’ve used in the past?

Me: Wow.

 

[Blue Mountain]

Now look at what I'm trying to get resolved. I'm trying to sign up for a large e-commerce site here in Canada. I've been on the support line with them three times for this issue and have not yet been successful.

The problem is I'm not receiving the emails they send out requesting confirmation of my sign-up request. Here's what I told them in the most recent chat session:

Several times now I've tried to sign up for EcommerceSite using the following two email addresses:

ecommerce-site-mail@myprivatedomain.ca
ecommerce-site-yiywiznj-mail@shaw.ca

I have never received the email you claim to have sent verifying my registration.

I've had more than one interaction with EcommerceSite's help line on this. To date the only substantive responses I've had are:

1. "Our investigation shows neither of these accounts are registered." That's hardly profound: they're not registered because I'm not receiving the registration emails!

2. "Try registering using a free email account from Gmail, Hotmail, or Yahoo."

These are NOT acceptable options.

- Gmail is owned by Google, which first and foremost is an advertising company. I loathe most advertising on the web and hate the cesspool that advertising and advertisers have turned the modern web into.

- Hotmail is owned by Microsoft, which has a long and sordid history of anticompetitive behaviour. Their Windows 10 product is crap and I hate the fact that Microsoft, one of the richest companies in the modern computer business, is using its operating system--which is supposed to be a utility product to provide an interface between the hardware and the user--for advertising. Basically, it's taken what's supposed to be a neutral platform and is using it to harvest user data and selling it for its own profit.

- Yahoo has had serious data breaches, at one point leaking the personal information of over a billion users! I wouldn't trust them to secure a ten ton concrete block chained to the seabed.

Ergo, I completely and absolutely refuse to do business with any of these companies.

Next points:

* I have used my @myprivatedomain.ca email address to sign up for eBay, Amazon, LinkedIn, and dozens of other accounts. Why is it that only EcommerceSite is having trouble with it?

* shaw.ca is a MAJOR Canadian web service provider and ISP. There is no excuse for EcommerceSite not being able to send email to them.

Final point: this is EMAIL. The standards date back 35 YEARS. Note that word, "standards." There are well documented ways for email to work. I am frankly amazed that EcommerceSite as a company is unable to deliver a simple email message using standard protocols.

Further, @myprivatedomain.ca is unfiltered. There is no spam or junk folder--all email sent there flows through to my email client, and it's the one doing the filtereing. There's no need to whitelist @ecommercesite.ca because it isn't on the blacklist. Correspondence I’ve had with with the web hosting provider hosting myprivatedomain.ca, shows no attempts by ecommerce-site.ca to connect to their email servers.

What came out of that last chat session?

Nothing, just like the other two.

Their support is terrible: they don't open tickets, they don't give case numbers, and they don't follow up. Is it any wonder I may get a little short with their personnel the next time I try to raise this issue?

I finally figured out what was happening.

I am very much a non-fan of of Google's and Facebook's pervasive presence and tracking on the web and very actively block them. In my HOSTS file I include the following lines:

# The Big Bad Bears of the internet (Google, Facebook, and Twitter)
0.0.0.0 facebook.com [url]www.facebook.com[/url] edge-mqtt.facebook.com
0.0.0.0 graph.facebook.com connect.facebook.net
0.0.0.0 mqtt.c10r.facebook.com star.c10r.facebook.com staticxx.facebook.com

0.0.0.0 google.com [url]www.google.com[/url]
0.0.0.0 clients1.google.com cse.google.com
0.0.0.0 docs.google.com dl.google.com googleads.g.doubleclick.net mail.google.com
0.0.0.0 plus.google.com safebrowsing-cache.google.com safebrowsing.google.com
0.0.0.0 sites.google.com sb-ssl.google.com ssl.google-analytics.com
0.0.0.0 [url]www.google-analytics.com[/url] [url]www.google.ca[/url] pagead2.googlesyndication.com
0.0.0.0 tpc.googlesyndication.com
0.0.0.0 [url]www.googletagmanager.com[/url]  [url]www.googletagservices.com[/url]

0.0.0.0 static-ads.twitter.com

The E-commerce site I was trying to sign up at uses Google's RECAPTCHA. In the site HTML, the first point of contact for that is www.google.com/recaptcha/api.js, which was blocked by my HOSTS file. As a result the CAPTCHA simply did not appear on the page and I had no idea it was there. Worse, the page accepted all my input, confirmed it had done so, and told me to wait for the confirmation email. But in fact the site had not accepted my input and had not sent the email. Only when I disabled my HOSTS file was I able to see the CAPTCHA, solve it, and complete the registration.

I used the site's Live Chat feature to bring this to their attention, asking the person on the chat to forward this as an issue to their web development team.