tag:blogger.com,1999:blog-46037507587358790142023-11-15T08:53:52.424-05:00Curiosities and SystemsRants, raves, and writings about being alive in the age of devops.Justin La Sottenhttp://www.blogger.com/profile/12564580388755776065noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-4603750758735879014.post-32841702150632928092013-10-30T14:51:00.000-04:002017-07-10T11:44:39.982-04:00Collectd + Bucky, lessons learned<span style="font-family: "arial" , "helvetica" , sans-serif;">I recently set out to have <a href="https://collectd.org/">collectd</a> capture metrics and pass them off to a centralized <a href="https://pypi.python.org/pypi/bucky/0.2.6">bucky</a> instance. Since I use CentOS 6.*, I installed </span><span style="font-family: "courier new" , "courier" , monospace;">collectd</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> 4.10.9-1 from the EPEL repo onto a test machine for sending metrics. This was the last stable release in the 4 series. I then installed </span><span style="font-family: "courier new" , "courier" , monospace;">python-bucky</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> 0.2.6-3 also from the EPEL repo, and </span><span style="font-family: "courier new" , "courier" , monospace;">collectd</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> 4.10.9-1 onto a test machine for storing metrics. I then connected the two machines together using the </span><span style="font-family: "courier new" , "courier" , monospace;">network</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> plugin. Together things were harmonious. Metrics were collected and stored just the way I wanted them.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The trouble began when I wanted to use a feature of collectd found only in the 5 series. I built collectd 5.4 from source and packaged nicely into RPMs, put the RPMs in my local yum repo and set off upgrading collectd on my test machine for sending metrics. I then modified the collectd *.conf files to account for the new version. Collectd started up smoothly and off to the races.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Watching bucky turned up some problems. It seemed that collectd wasn't sending metrics for certain plugins. I had configured collectd to send metrics for 7 plugins, yet I was only receiving about 4 of them. It also seemed like collectd was sending the metrics was receiving at seemingly random intervals. This sent me into a tizzy.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I was stumped. Did I mess up the compile? Did I miss something in the RPM build? I went over those steps again to see if I forgot anything, I couldn't find anything. I popped onto IRC and asked if anyone had any troubles like this before. Not much help there, but a key point came out that set me on the right path. Someone suggested using the </span><span style="font-family: "courier new" , "courier" , monospace;">CSV</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> plugin rather than the </span><span style="font-family: "courier new" , "courier" , monospace;">network</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> plugin to see if the collectd collectors were indeed working correctly. This proved that the collectors were in fact working and collecting metrics properly. So what could it be?</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I fired up <a href="http://www.tcpdump.org/tcpdump_man.html">tcpdump</a> on both machines to see if the packets were being sent and received. They were. I then changed my tcpdump strategy to so that it would decode the UDP packets. I could see all the metrics being sent. I noticed that the order of the metrics in the packets changed with each packet sent. Bucky would read the packet, find any metrics until it came across one of the "bad metrics" and then cease processing the pay load. When the packet pay load started with a "bad metric" nothing would be processed and would look like bucky never received anything.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I started pouring through bucky's code thinking that perhaps the collectd <a href="https://collectd.org/wiki/index.php/Binary_protocol">network binary protocol </a>had changed and bucky simply wasn't updated to handle it. After some time, I came to the conclusion that, no, bucky was fine. Its protocol decoding was good. What the heck else could it be?</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Then it hit me, while looking at the bucky source code, I saw reference to the collectd </span><span style="font-family: "courier new" , "courier" , monospace;">types.db</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> which comes with collectd. <a href="http://collectd.org/documentation/manpages/types.db.5.shtml">Types.db</a> is a specification for the metrics collected by collectd. It basically describes each metric and how it's data should be handled.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Well, bucky was installed on a machine that had collectd 4.10.9-1 installed, not the newer 5.4. I opened up the types.db for both 4.10.9-1 and for 5.4 and saw that the specification for the metrics I was losing, the "bad metrics", had changed. Doh!</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I upgraded the collectd on the </span><span style="font-family: "arial" , "helvetica" , sans-serif;">test machine for storing metrics, which upgraded the types.db file.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> Restarted bucky so that it could read the new types.db. VoilĂ ! Metrics started to pour in again!</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Lesson learned. Bucky is highly dependent on the version of collectd both installed elsewhere as well as the local install. They should/must match otherwise some metrics could be misinterpreted, or worse, skipped entirely.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Hopefully this post will help someone having the same problem!</span>Justin La Sottenhttp://www.blogger.com/profile/12564580388755776065noreply@blogger.com0tag:blogger.com,1999:blog-4603750758735879014.post-29985182513670254262013-10-30T11:47:00.000-04:002013-10-30T11:47:34.947-04:00Graphite Logins, part of the story<span style="font-family: Arial, Helvetica, sans-serif;">There seems to be a lack of information, or rather, a disconnect of information regarding graphite and graphite logins. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The problem, seems to me, is that graphite indicates that it has it's own logins, but doesn't seem to give the ability to create them. During installation, you do indeed create one superuser, and you can indeed log in with it, but, how do you create more logins for anyone other than the admin?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Enter Django. I knew graphite was built on top of Django, however, having no prior experience with Django, I had no idea how they were coupled and what Django brought to the table. Well it turns out that graphite, being a Django app, has an admin panel. You can find the panel located at:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">http://www.example.com/graphite/admin/</span></blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;">or some variation on the above URL. The key part is the suffix of </span><span style="font-family: Courier New, Courier, monospace;">/admin/.</span><span style="font-family: Arial, Helvetica, sans-serif;"> To get into this panel for the first time, you can use the superuser you created at graphite installation time. From here it's just a few simple clicks to add groups and users. It's also give you handy access to saved dashboards in case you want to hand modify them.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">If you've forgotten the master user/pass you created at installation time, no worries, we can create a new one. For this, though, we'll need to reach for the command line (If there is another way of doing this please let me know). First we need to locate the Django management script called </span><span style="font-family: Courier New, Courier, monospace;">manage.py</span><span style="font-family: Arial, Helvetica, sans-serif;">. I'm not going to presume to know how you installed graphite, you could have installed it from source, or like me, from a repo. What I did was run the command:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">locate manage.py</span></blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;">and picked out the </span><span style="font-family: Courier New, Courier, monospace;">manage.py</span><span style="font-family: Arial, Helvetica, sans-serif;"> that pertained to graphite. In my case:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">/usr/lib/python2.6/site-packages/graphite/manage.py</span></blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;">If you followed the default installation guide from the graphite docs it might be located at</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">/opt/graphite/webapp/graphite/manage.py</span></blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;">Now that we've found the correct </span><span style="font-family: Courier New, Courier, monospace;">manage.py</span><span style="font-family: Arial, Helvetica, sans-serif;"> we can go ahead and create a superuser via the command:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">manage.py createsuperuser</span> </blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;">This will prompt you for username, email, and password. Once you are done, you'll have a new superuser that you can use to log into the graphite admin panel outlined above.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Further Reading:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">The Django project has loads of good documentation and the most relevant to this post is the <a href="https://docs.djangoproject.com/en/1.6/topics/auth/default/">Django Authentication System</a> where you can find out about creating users via the Python API, and a whole lot more.</span>Justin La Sottenhttp://www.blogger.com/profile/12564580388755776065noreply@blogger.com0