{"id":7190,"date":"2023-11-13T21:07:26","date_gmt":"2023-11-13T20:07:26","guid":{"rendered":"http:\/\/blog.bart.sk\/en\/?p=7190"},"modified":"2024-01-25T12:39:50","modified_gmt":"2024-01-25T11:39:50","slug":"monitoring-applications-nutshell-part-2-need-grafana-alerting","status":"publish","type":"post","link":"https:\/\/blog.bart.sk\/en\/monitoring-applications-nutshell-part-2-need-grafana-alerting\/","title":{"rendered":"Monitoring Applications in a Nutshell &#8211; Part 2: Why Do We Need Grafana Alerting?"},"content":{"rendered":"<p><b>In the first part of the series of articles entitled <\/b><a href=\"https:\/\/blog.bart.sk\/en\/monitoring-applications-nutshell-part-1-monitor\/\"><b>Monitoring applications in a nutshell<\/b><\/a><b>, you learned about a presentation tool that displays and analyzes the data necessary for monitoring our web application. In this sequel, we&#8217;ll detail how we automate tracking to minimize predictable errors and prevent possible app crashes.<\/b><\/p>\n<p><span style=\"font-weight: 400;\">The aim of our monitoring is to:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><b>Centralize both server and client workload records<\/b><span style=\"font-weight: 400;\"> &#8211; We use <\/span><b>Prometheus <\/b><span style=\"font-weight: 400;\">and <\/span><b>Cloudwatch<\/b><span style=\"font-weight: 400;\"> tools to manage this<\/span> <span style=\"font-weight: 400;\">data.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Monitor the obtained records in detail<\/b><span style=\"font-weight: 400;\"> &#8211; Grafana allows us to visualize them based on the data from the first target.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Ensure the possibility of process automation <\/b><span style=\"font-weight: 400;\">&#8211; <\/span><b>Alerting<\/b><span style=\"font-weight: 400;\"> is one of the tools for automating simple monitoring<\/span><b>.<\/b><\/li>\n<\/ol>\n<h2><b>What is Grafana alerting and how to set it up?<\/b><span style=\"font-weight: 400;\">\u00a0<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">In the<\/span><a href=\"https:\/\/blog.bart.sk\/monitoring-aplikacii-v-kocke-1-cast-co-vsetko-sledujeme-preco\/\"><span style=\"font-weight: 400;\"> previous article<\/span><\/a><span style=\"font-weight: 400;\">, we defined the most important aspects of performance, load, process, memory and disk monitoring. Data from these monitorings are available on the basis of the aforementioned Prometheus, which together with Cloudwatch is key for alerting.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In short:\u00a0<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>Prometheus: <\/b><span style=\"font-weight: 400;\">An open-source system that uses a pull model to collect metrics. This means that the Prometheus server regularly calls endpoints in the monitored services and receives metrics from them.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>CloudWatch<\/b><span style=\"font-weight: 400;\">: An Amazon service that uses a push model (although it also supports pull for some services). Services and applications send their metrics to CloudWatch and visualizations are created based on data from these sources.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">These services send their data to Grafana, which visualizes them. The result looks something like this:<\/span><\/p>\n<figure id=\"attachment_11024\" aria-describedby=\"caption-attachment-11024\" style=\"width: 1600px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/07\/Untitled-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11024 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/07\/Untitled-2.png\" alt=\"\" width=\"1600\" height=\"737\" \/><\/a><figcaption id=\"caption-attachment-11024\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Data visualization in Grafana<\/span><\/i><\/figcaption><\/figure>\n<p><b>And what do the two dashed lines mean? This is part of automatic monitoring, which can be set as follows:<\/b><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">In Grafana, you<\/span><b> define the Data Source<\/b><span style=\"font-weight: 400;\"> and process it into the form you need.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Subsequently, the <\/span><b>data must be cleaned<\/b><span style=\"font-weight: 400;\"> or edited. NaN or NULL values are usually omitted. However, it&#8217;s necessary to consider whether these values aren&#8217;t necessary for the correct evaluation of your conditions and whether their removal won&#8217;t cause an unwanted ignoring of a potential error.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Finally, you<\/span><b> specify concrete conditions above the output value<\/b><span style=\"font-weight: 400;\"> when the alert goes into one of the following states:<\/span>\n<ul>\n<li style=\"font-weight: 400;\"><b>Normal: <\/b><span style=\"font-weight: 400;\">No risk.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Pending: <\/b><span style=\"font-weight: 400;\">A regular check-up of the conditions is under way and in case one of them is met, the alert is moved to the Firing state.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Firing: <\/b><span style=\"font-weight: 400;\">Alerts administrators, and possibly the client, about a potential problem.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p><b>It&#8217;s very important to determine the conditions of automatic monitoring correctly so that as few false alerts as possible<\/b> <b>come out<\/b><span style=\"font-weight: 400;\">, which can cover the really important ones. In the case of frequent repeated warnings or false &#8220;alarms&#8221;, alerts may begin to be considered untrustworthy. Subsequently, there&#8217;s a risk that the administrator will ignore them, and ultimately they&#8217;ll lose their original purpose. Thus, the process of defining these conditions is usually not a one-time process, but a long-term one in order to fine-tune all the details.\u00a0<\/span><\/p>\n<p><b>At Crossuite, we currently have alerts divided into 3 different states:<\/b><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><b>Warning indicates a potential problem.<\/b><span style=\"font-weight: 400;\"> The user can set this alert according to different criteria or thresholds. The Warning is displayed to us when the data is returned and modified before the criticality limit we set in advance. If this potential problem isn&#8217;t addressed, it could cause a web application outage in the near future.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Critical is an issue that was previously reported as a Warning but wasn&#8217;t resolved, <\/b><span style=\"font-weight: 400;\">or an<\/span><b> issue caused by a sudden error<\/b><span style=\"font-weight: 400;\">. Both types cause a potential blocking aspect for the user and need to be addressed immediately.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Business alerts <\/b><span style=\"font-weight: 400;\">are alerts<\/span> <span style=\"font-weight: 400;\">captured directly based on user interaction.<\/span><\/li>\n<\/ol>\n<h2><b>Example: Alert setting &#8211; running out of disk space<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">One of the very important alerts is disk usage.<\/span><b> If the drive runs out, it can have various consequences, such as data loss, service outage or performance problems.<\/b><span style=\"font-weight: 400;\"> Since none of these consequences are acceptable to the end user, we closely monitor disk usage along with many other metrics, of course.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In the following picture, I cut out a few alerts from our automatic monitoring. As you can see, <\/span><b>Disk usage isn&#8217;t a common problem, as even we have this alert in the Firing state<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<figure id=\"attachment_11256\" aria-describedby=\"caption-attachment-11256\" style=\"width: 1289px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11256 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-2.png\" alt=\"\" width=\"1289\" height=\"848\" \/><\/a><figcaption id=\"caption-attachment-11256\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Sample alerts from the Crossuite project<\/span><\/i><\/figcaption><\/figure>\n<h2><b>How to set up such an alert?<\/b><\/h2>\n<p><b>We define the data source for alerts using the query language for Prometheus:<\/b><\/p>\n<figure id=\"attachment_11262\" aria-describedby=\"caption-attachment-11262\" style=\"width: 1489px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-3_blur.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11262 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-3_blur.jpg\" alt=\"\" width=\"1489\" height=\"270\" \/><\/a><figcaption id=\"caption-attachment-11262\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Definition of data sources<\/span><\/i><\/figcaption><\/figure>\n<p><span style=\"font-weight: 400;\">What does this command do? In layman&#8217;s terms<\/span><i><span style=\"font-weight: 400;\">, <\/span><\/i><span style=\"font-weight: 400;\">we can translate it as follows:<\/span><i><span style=\"font-weight: 400;\"> &#8220;Tell me how much available space in gigabytes there is in each file system belonging to the group \u2018bart.sk\u2019, but don&#8217;t take into account those that:<\/span><\/i><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><i><span style=\"font-weight: 400;\">are on certain devices,\u00a0<\/span><\/i><\/li>\n<li style=\"font-weight: 400;\"><i><span style=\"font-weight: 400;\">have certain types of file systems,<\/span><\/i><\/li>\n<li style=\"font-weight: 400;\"><i><span style=\"font-weight: 400;\">are connected in certain places.<\/span><\/i><\/li>\n<\/ul>\n<p><i><span style=\"font-weight: 400;\">For a simpler overview, round the values to the thousandths of a gigabyte.&#8221;<\/span><\/i><\/p>\n<p><b>Explanation of specific metrics:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>node_filesystem_avail_bytes:<\/b><span style=\"font-weight: 400;\"> This metric contains the number of bytes that are available on the file system.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>{group=\u201cbart.sk\u201c,&#8230;}:<\/b><span style=\"font-weight: 400;\"> Filters that determine which file systems should be included or excluded from the results. For example, group= \u201cbart.sk\u201c means that we only look at file systems that belong to the \u201cbart.sk\u201c group.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>device!~\u201c\/dev\/mapper\/vg_pnapi1-opt\u201d<\/b><span style=\"font-weight: 400;\"> and similar filters starting with device!~ define the inclusion\/exclusion of certain devices (disks or partitions) determined by the string after !~.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>mountpoint !~ \u201e\/run.*\u201c<\/b><span style=\"font-weight: 400;\"> and other mountpoint filters indicate that the results shouldn&#8217;t contain certain connected points (places where disks are accessible in the file system) based on a pattern or an exact path.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Fstype!~\u201ctmpfs\u201c<\/b><span style=\"font-weight: 400;\"> specifies the inclusion\/exclusion of file systems of the \u201etmpfs\u201c type, which are temporary file systems stored in RAM.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>mountpoint!=\u201c\/var\/www\/data\/private-aws\u201d<\/b><span style=\"font-weight: 400;\"> and others indicate the inclusion\/exclusion of specifically connected points from the results.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>\/ (1024 * 1024 * 1024):<\/b><span style=\"font-weight: 400;\"> This part of the formula converts the value from bytes to gigabytes because 1024 bytes is one kilobyte, 1024 kilobytes is one megabyte, and 1024 megabytes is one gigabyte.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>,0.001):<\/b><span style=\"font-weight: 400;\"> This section rounds the resulting value to thousandths of gigabytes to make the values clearer.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">The query will then<\/span><b> display the data to us in the same or similar format as we&#8217;ve defined for each file system<\/b><span style=\"font-weight: 400;\">:<\/span><\/p>\n<figure id=\"attachment_11254\" aria-describedby=\"caption-attachment-11254\" style=\"width: 1486px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-4.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11254 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-4.png\" alt=\"\" width=\"1486\" height=\"310\" \/><\/a><figcaption id=\"caption-attachment-11254\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Sample of acquired data<\/span><\/i><\/figcaption><\/figure>\n<p><span style=\"font-weight: 400;\">In the next step, if necessary, <\/span><b>we can clean non-numeric, undefined or zero values from this data<\/b><span style=\"font-weight: 400;\">.\u00a0<\/span><\/p>\n<figure id=\"attachment_11253\" aria-describedby=\"caption-attachment-11253\" style=\"width: 643px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-5.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11253 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-5.png\" alt=\"\" width=\"643\" height=\"98\" \/><\/a><figcaption id=\"caption-attachment-11253\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Filtration of NaN and Null values<\/span><\/i><\/figcaption><\/figure>\n<p><span style=\"font-weight: 400;\">Then we<\/span><b> define the evaluation conditions on the basis of which this alert will be assigned to defined states<\/b><span style=\"font-weight: 400;\"> (warning, critical, business). We&#8217;ve defined the free space using expression that&#8217;ll return the value in percentage:<\/span><\/p>\n<p><b><i>\u00a0${reduced_filesystem_available_size} \/${reduced_filesystem_size}*100<\/i><\/b><\/p>\n<figure id=\"attachment_11251\" aria-describedby=\"caption-attachment-11251\" style=\"width: 1324px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-6_blur.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11251 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-6_blur.jpg\" alt=\"\" width=\"1324\" height=\"494\" \/><\/a><figcaption id=\"caption-attachment-11251\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Definition of expressions<\/span><\/i><\/figcaption><\/figure>\n<p><b>In this case, we&#8217;ve set 2 conditions:<\/b><\/p>\n<p><b>Warning condition:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If the value is less than 15%, the alert enters the <\/span><b>PENDING<\/b><span style=\"font-weight: 400;\"> state<\/span><b>.<\/b><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If this state persists longer than the defined T time, then the alert goes to the <\/span><b>WARNING<\/b><span style=\"font-weight: 400;\"> state<\/span><b>. <\/b><span style=\"font-weight: 400;\">\u00a0<\/span><\/li>\n<\/ul>\n<p><b>Critical condition<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If the value is less than 10%, the alert enters the <\/span><b>PENDING<\/b><span style=\"font-weight: 400;\"> state<\/span><b>.<\/b><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If this state persists longer than the defined T time, then the alert goes to the <\/span><b>CRITICAL<\/b><span style=\"font-weight: 400;\"> state<\/span><b>.<\/b><\/li>\n<\/ul>\n<p><b>In this case, the Critical condition is always superior to the Warning condition.<\/b><\/p>\n<p><b>The resulting alert is then sent to the space defined within the Grafana settings<\/b><span style=\"font-weight: 400;\"> using Webhook. For us, it&#8217;s Google Chat.\u00a0<\/span><\/p>\n<p><b>The form and structure of the alert are defined using the templating language:<\/b><\/p>\n<figure id=\"attachment_11252\" aria-describedby=\"caption-attachment-11252\" style=\"width: 1170px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-7.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11252 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-7.png\" alt=\"\" width=\"1170\" height=\"362\" \/><\/a><figcaption id=\"caption-attachment-11252\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Notification structure<\/span><\/i><\/figcaption><\/figure>\n<p><b>Subsequently, individual values can be defined for the template using Annotations and Variables:<\/b><\/p>\n<figure id=\"attachment_11250\" aria-describedby=\"caption-attachment-11250\" style=\"width: 892px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-8_blur.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11250 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-8_blur.jpg\" alt=\"\" width=\"892\" height=\"604\" \/><\/a><figcaption id=\"caption-attachment-11250\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Definition of Annotations and Variables<\/span><\/i><\/figcaption><\/figure>\n<p><b>The specific notification of lack of disk space then looks like this:<\/b><\/p>\n<figure id=\"attachment_11249\" aria-describedby=\"caption-attachment-11249\" style=\"width: 509px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-9_blur.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11249 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-9_blur.jpg\" alt=\"\" width=\"509\" height=\"490\" \/><\/a><figcaption id=\"caption-attachment-11249\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Alert preview<\/span><\/i><\/figcaption><\/figure>\n<h2><b>So what happens next?<\/b><\/h2>\n<p><b>Of course, each alert should be linked to some graphical visualization<\/b><span style=\"font-weight: 400;\"> thanks to which it&#8217;s possible to examine the problem more closely. Here we come to the question from the previous article &#8211; what are the dotted lines in the graphs? These are the exact points where the alert has identified a problem. <\/span><b>In this way, we can find out what preceded the situation and define the conditions that led to it.<\/b><span style=\"font-weight: 400;\"> The graph in which the alert should identify it is determined using Summary and Annotations with Dashboard UID and Panel ID items.\u00a0<\/span><\/p>\n<figure id=\"attachment_11248\" aria-describedby=\"caption-attachment-11248\" style=\"width: 1129px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-10_blur.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-11248 size-full\" src=\"https:\/\/blog.bart.sk\/wp-content\/uploads\/2023\/11\/pasted-image-0-10_blur.jpg\" alt=\"\" width=\"1129\" height=\"564\" \/><\/a><figcaption id=\"caption-attachment-11248\" class=\"wp-caption-text\"><i><span style=\"font-weight: 400;\">Data visualization in Grafana<\/span><\/i><\/figcaption><\/figure>\n<p><span style=\"font-weight: 400;\">We&#8217;re at the end of the second installment of the Monitoring in a nutshell series and an <\/span><b>attentive reader certainly didn&#8217;t miss that I haven&#8217;t explained what our Business alerts are for<\/b><span style=\"font-weight: 400;\">. Don&#8217;t worry, you&#8217;ll learn all about them and how they relate to Loki logs in the next part. See you then, readers \ud83d\ude42<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"In the first part of the series of articles entitled Monitoring applications in a nutshell, you learned about&hellip;","protected":false},"author":30,"featured_media":7282,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","csco_display_header_overlay":false,"csco_singular_sidebar":"","csco_page_header_type":""},"categories":[199],"tags":[499,269,498,334,314,500],"_links":{"self":[{"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/posts\/7190"}],"collection":[{"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/users\/30"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/comments?post=7190"}],"version-history":[{"count":2,"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/posts\/7190\/revisions"}],"predecessor-version":[{"id":7193,"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/posts\/7190\/revisions\/7193"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/media\/7282"}],"wp:attachment":[{"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/media?parent=7190"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/categories?post=7190"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.bart.sk\/en\/wp-json\/wp\/v2\/tags?post=7190"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}