Episode 180 – JSON All The Things!

In Episode 180, Ben and Scott talk through a recent experience Scott had when working with service principals in Azure AD for use with Azure Kubernetes Service. They also get into some of the fun that can be had with parsing JSON from the command line with jq and JMESPath. Transcript Email Download New Tab - [Ben] Welcome to Episode 180 of the Microsoft Cloud IT Pro Podcast recorded live on May 29 2020. This is a show about Microsoft 365 and Azure from the perspective of IT pros and end users where we discuss the topic or recent news and how it relates to you. In this episode, Scott and Ben talked through a recent experience Scott had when working with service principals in Azure AD for use with Azure Kubernetes Service and then wrap up talking a little bit about Parsing JSON in the command line. Hey, Scott, we're recording Episode 180 today. - [Scott] 180, amazing. - [Ben] It really is. And you know what other mark we crossed recently, 'cause we don't talk about this much. 400,000 downloads over the lifetime of the podcast. - [Scott] Nice, I gotta go update the About page. - [Ben] You should. We should actually make some updates around that. So we have, it has been going on three years now. Because we record extra at events. I can't remember when we started this, but it was about this time three years ago. Which means I have talked to you every week for three years. - [Scott] And you're better for it. I keep telling you that, but you don't believe me. - [Ben] I would not be the same man I am today had it not been for you, Scott. - [Scott] Boy, let's see if I can teach you some stuff today. Maybe you too could teach me some stuff. - [Ben] If we talk about AKS, like you have suggested you would like to because you have been living in the world of AKS. Not to be confused with ACKS when you're talking about networking. Bad joke. - [Scott] No, there's a TCP joke in there somewhere. - [Ben] Yes, you will probably be teaching me more because AKS is not something that I have done much with or have much knowledge in. So would you like to talk AKS today, Scott? - [Scott] Yeah, we can talk about AKS. I have a bit of a personal question first though. So my wife and I have been going back and forth on this one. Do you ever have dreams about work? Like, do you get so involved in your work that you go to bed at night and you dream about it? - [Ben] Not a lot. I think I've done it once or twice, but it is not a regular occurrence. However, based on your Facebook status, I thought that was a joke. So is your Facebook status actually happened to you the other night? It was at twitter. - [Scott] It's been happening to me on the regular lately. So I am either far too close to my work, or my brain is just warping in a very strange way. So last night, I was watching TV and fell asleep on the couch. And I'd been spending a little bit of time in the morning running through and playing within AKS deployment. And one of the things you end up doing in AKS quite a bit after it's deployed or heck even, while it's deploying, especially when you're working with Azure CLI or things like that is you're gonna be playing a lot with JSON objects. When you're in bash, you want to find better ways to play with JSON. Because you don't wanna be potentially just manipulating raw strings or not having things like access to logical operators for doing selects of given nodes in a JSON document or anything like that, right? You just need a way to like prettify it and make it usable to you, so that you can query within it and kind of figure things out. So one of the tools that I found that I like to do that is jq. So that's a lightweight and flexible command line JSON processor. Think like I said for JSON data. And, you're on the right path for what it does. Certainly, there's a lot of tooling built into other things like Azure CLI supports queries with JMESPath. If you're looking like I was doing a bunch of stuff with, like you said, AKS over the last couple weeks. One of the things you end up doing there is using kube-control or kube-cuddle, whatever you want to call it, quite a bit to, you're always querying for services, deployments, nodes, pods, things like that. And there's an operator where you can either get your output back from kube-control as JSON, or you can use what they've built in for JSONPaths, which uses a query language that's like what's in JMESPath, but not all the way there, which is like what's in jq, but not all the way there. So my brains been really worked around these things. So I spent, I don't know, maybe like 30-45 minutes in the morning, trying to figure out this one jq query or I would have been happy if it was a JSONPath query or anything else that I could have gotten it into. When you're in Kubernetes, and you're doing something like, say you do kube-control, get nodes or describe nodes, whatever you wanna do to get your data back. Some providers like Microsoft, when they do things like label a node or label a pod with their annotations, they'll go ahead and label them in a way that some JSON parsers, particularly like jq don't like. They will have characters in them that just can't be parsed from a bash command line without a bunch of escaping, or other things otherwise. So something like a forward slash, or an equal sign, just weird things through bash off in general, and then you end up in string processing land and all these kinds of things. - [Ben] So what you're saying is, Microsoft did not follow conventions around something that has been a standard. Not to bash on Microsoft. - [Scott] So jq is totally an external tool. - [Ben] So they don't follow certain standards. Microsoft's not just ignoring standards here. - [Scott] I can't say that I love that, a million of their annotations have things like forward slashes in them which some parsers treat as a new node and not part of just a key value pair, things like that. - [Ben] Got it. So is there a standard around how this should all be laid out when you go look at these parsers and stuff, not to go down a rabbit hole? - [Scott] Well, I mean, it's just JSON. So they should be able to parse JSON. But now then you get into, I'm trying to parse from bash, or just a Linux shell. So whether it was zsh, ksh, or bash, whatever it happened to be, they all have their own rules for string processing from the command line, and then you're trying to figure out the right way to pass strings into things to get it all wired up and where it needs to be. Anyway, I was doing some stuff trying to get down to, I was just trying to write a simple query that retrieved nodes in a node pool. And it gave me the node pool name 'cause it's a cluster with a bunch of different node pools, and then I needed an annotation off of there for the failure domains. So when you create multiple node pools within AKS, you can also take those node pools and push them out to availability zones if you want to. So you can have each node in an AZ. So really the way to figure out where your node sits, is doing something like querying for the failure domain. So it's failure-domain.beta.kubernetes.io/zone, and then that'll give you the name of the zone you're in. And then you can pull out the node name or metadata.labels.agentpool gives you the node name and things like that. So you end up doing all this weird stuff. All I wanted to do was, I wanted. Well, I was first trying to just do a select statement. So with most selects, and you go look at jq's documentation, you're like, all right, select equals this. That makes sense, right? As a nice operator to do equality tax. It also does greater than and less than, but I didn't want to do any of those things, I wanted to do basically a like, or a match, like you might be used to inside of PowerShell. Or you do like, Where clause with a wildcard or something like that. - [Ben] So, essentially, like in CONTAINS. - [Scott] Yeah, exactly. But it took 45 minutes for CONTAINS to click in my head. So I was diving through the jq documentation, which isn't like the greatest thing, hey, it's okay. but I'm sitting there like, "All right, so there's not a practical example "for the one thing I wanna do." And I should have just rung you up, 'cause you would have said, "Hey, you're trying to do a CONTAINS," and then I would have just done a find on the operator page, and the function page and said, "Look, there's a CONTAINS function." That will help me a whole bunch and bring it back. So I've been living like this weird nightmare of JSON and CLI and all sorts of fun things like that. - [Ben] So to bring that full circle that you actually dream about that then last night while you were sleeping on the couch? - [Scott] Yeah, no, totally had a dream about browsing through the jq documentation. And looking for select and then it was just awesome. - [Ben] We need to find you some hobbies or we need to get you doing more work so that you're way more tired when you go to bed and you don't dream about it. - [Scott] Yeah, no, it's, I agree something to do there. - [Ben] As IT professionals in the Cloud era, sometimes that feels like we don't speak the same language as the rest of the organization. So when stakeholders from finance or other departments start asking about a specific project or teams Azure costs, they don't always realize how much work is involved in obtaining that information, sifting through cluttered CSVs and a complex mess of metadata in order to manually create custom views and reports. It's a real headache. On top of helping you understand and reduce your organization's overall Azure spend, ShareGate overcast, lets you group resources into meaningful cost hubs and map them to real world business scenarios. This way, you can track costs in the way that makes most sense with your corporate structure....