#rabbitmq IRC Log


IRC Log for 2011-11-09

Timestamps are in GMT/BST.

[9:17] <snack_> antares_working: ping
[9:37] <antares_working> snack_: pong
[9:40] <snack_> antares_working: I'm still getting "undefined method `final?' for #<AMQ::Protocol::BodyFrame:0xf50f7f8>"
[9:40] <snack_> in amq-client-0.8.5/lib/amq/client/async/adapter.rb:655
[9:40] <antares_working> snack_: that is because framing is still not per-channel
[9:40] <snack_> not sure what causes it, but it freezes up processing on some of my workers
[9:41] <snack_> antares_working: how can we solve this?
[9:41] <antares_working> snack_: I can explain where the code that handles framing is and what needs to be changed, you can change it, run tests and send a pull request
[9:41] <snack_> antares_working: ok
[9:42] <antares_working> snack_: https://github.com/ruby-amqp/amq-client/blob/master/lib/amq/client/adapter.rb#L490
[9:43] <antares_working> snack_: basically, right now we detect whether frame sequence is final by keeping a list of frames that does not take channels into account. We need to do that.
[9:43] <antares_working> snack_: it is not a lot of work and nearly any issue will result in test failures all over the place. Should I continue explaining?
[9:44] <snack_> so... there is some communication that is not inside a channel, for example, setting up a channel, correct?
[9:45] <antares_working> correct
[9:45] <antares_working> and some frames do carry payload while other do not
[9:46] <antares_working> basic.publish and basic.return do, other methods don't
[9:46] <antares_working> in amq-protocol, frame classes implement #final? and #has_content? methods, I think Java client does things exactly the same way
[9:46] <snack_> so why doesn't BodyFrame include it?
[9:47] <antares_working> snack_: also, if basic.pulish payload is blank, body frames are not sent (only basic.publish and the metadata header)
[9:48] <antares_working> snack_: I think it would be easier to understand from the code than to explain here: https://github.com/ruby-amqp/amq-protocol/blob/master/lib/amq/protocol/client.rb
[9:48] <antares_working> snack_: that file is generated and uses rabbitmq-codegen (so templating is in Python) that many other clients use
[9:49] <antares_working> but I doubt there will be need to change anything in amq-protocol
[9:50] <snack_> see I'm not familiar with the protocol. I just know I get this exception fairly regularly and it stops my processing, which is pretty critical to keep running
[9:50] <snack_> sounds like a major bug to me
[9:52] <antares_working> snack_: I don't have the time to work on this issue till Friday
[9:53] <antares_working> although I will try. So I don't know what other options we have.
[9:53] <snack_> I'm just not sure why BodyFrame doesn't implement final?
[9:54] <antares_working> there are 3 frame types
[9:54] <antares_working> method frames (that encode operation), headers frames (message metadata) and body frames (message content chunks)
[9:54] <snack_> ok
[9:55] <snack_> like payload?
[9:56] <antares_working> message payload is not sent all at once
[9:57] <antares_working> it is chunked (although chunk size is often quite large, like 128K)
[9:57] <antares_working> when you send a message using basic.publish what is sent is a sequence of frames
[9:58] <snack_> method, headers, body frames
[9:58] <antares_working> [method frame for basic.publish] [message header (content length and so on)][0 or more content chunks]
[9:58] <snack_> ok
[9:59] <antares_working> looking at the code I am not entirely sure what was the idea behind #final? but frame classes in amq-protocol provide certain information like whether methods carries content and so on
[9:59] <antares_working> body frames, however, don't know much about the entire message
[10:01] <snack_> some fo the frame types have the final? hardcoded to true.... so those are final frames based on the protocol spec, right?
[10:02] <snack_> as for the body frame, there must be some way to determine this based on something that comes over the wire
[10:03] <antares_working> snack_: message content length is passed in header frames
[10:04] <antares_working> snack_: I have a feeling that the #final? part can be removed
[10:04] <antares_working> snack_: ok, let me try it
[10:04] <snack_> hmmm
[10:04] <snack_> first_frame.final? || (first_frame.method_class.has_content? && content_complete?(frames[1..-1]))
[10:04] <snack_> ... because the other expressions are also ways to check finality?
[10:05] <antares_working> clause on the right hand side makes sense
[10:05] <snack_> right
[10:05] <antares_working> the left one does not
[10:05] <snack_> sounds redundant
[10:05] <antares_working> let me try removing it
[10:05] <antares_working> test suites will catch nearly any issue in that area
[10:05] <snack_> I'll check, too.
[10:07] <antares_working> class AMQ::Protocol::Frame
[10:07] <antares_working> def final?
[10:07] <antares_working> true ####### HACK for testing, implement & move to amq-protocol!
[10:07] <antares_working> end
[10:07] <antares_working> end
[10:07] <antares_working> sigh
[10:07] <antares_working> the person who was working on amq-protocol initially does not use amqp so he left a lot of edge cases uncovered and some cruft like this
[10:08] <snack_> ahh
[10:08] <antares_working> so yes, it can be removed I guess
[10:08] <snack_> tests pass on your side?
[10:08] * antares_working shakes fists at people who develop libraries but do not use them
[10:08] <snack_> heh
[10:08] <antares_working> snack_: I haven't run them after the change yet
[10:08] <snack_> whereas I use it, but don't develop it. :/
[10:10] <antares_working> I see failures :/
[10:10] <snack_> no rake task for specs?
[10:11] <snack_> er, not rakefile at all?
[10:12] <antares_working> bundle exec rspec -c spec
[10:12] <antares_working> there are no other tasks we would need a Rakefile for
[10:12] <snack_> ok
[10:13] <snack_> I wasn't aware of rspec's -c
[10:16] <antares_working> snack_: I think I understand the intent now
[10:16] <antares_working> snack_: let me look something up in the spec
[10:17] <snack_> meanwhile I think I found another gray hair
[10:18] <snack_> grr, I don't have a rabbitmq daemon on localhost :/
[10:19] <antares_working> snack_: so connection failed because of that?
[10:19] <snack_> in the rspec, yeah
[10:19] <antares_working> snack_: there is also a ./bin/* script that sets up vhosts & users
[10:19] <snack_> can I use a remote rabbitmq for rspec?
[10:20] <snack_> I saw that
[10:20] <antares_working> after every rabbitmq upgrade locally I forget to run it and wonder why some tests fail :)
[10:20] <antares_working> snack_: I am not sure host is configurable for amq-client. amqp gem used to allow that, but probably not any more
[10:20] <antares_working> nobody really does that among developers. And since about May there has been primarily one developer, myself.
[10:20] <snack_> no worries. I can install it
[10:21] <antares_working> snack_: so on, but I figured out what we need to do.
[10:21] <snack_> antares_working: I appreciate everything. Rabbitmq/amqp fit my application very well... when they work. :P
[10:23] <antares_working> snack_: I *think* I fixed it
[10:23] <antares_working> snack_: do you guys use Bundler?
[10:23] <antares_working> for your app in question?
[10:24] <snack_> antares_working: yes
[10:25] <antares_working> snack_: then I'll ask you to switch to :git for a day or so to test it
[10:25] <antares_working> for amq-protocol and amq-client
[10:26] <snack_> hmm, I wonder how that will affect our releases.... I can test it
[10:26] <antares_working> well, alternatively I can release 2 gems
[10:26] <antares_working> without announcing them :(
[10:26] <snack_> I "bundle" all the gems under vendor/ in the release cycle
[10:26] <antares_working> because release announcements always take half an hour to write…
[10:26] <snack_> no, no need to do that
[10:27] <antares_working> snack_: then lets push 2 new gems, vendoring + gems from git do not work well
[10:27] <snack_> well
[10:27] <antares_working> snack_: that's ok, I am already cutting a new amq-protocol release
[10:28] <snack_> I think I can make this work.... what are the branches?
[10:28] <snack_> ok. :P
[10:33] <antares_working> snack_: ok, try upgrading to amq-protocol 0.8.4 and see if your tests pass and so on
[10:33] <antares_working> snack_: I will cut amq-client that depends on that version as soon as you confirm that we did not break anything obvious
[10:36] <snack_> will do. the only change is in amq-protocol?
[10:36] <antares_working> snack_: no but for your case upgrading it is sufficient
[10:39] <antares_working> snack_: amq-client 0.8.6 is now released, too
[10:40] <antares_working> snack_: fortunately it was not the issue I thought you were hitting. That would require more work. Hopefully your consumers now will run happily for months and months :)
[10:43] <snack_> I'm setting up a test right now...
[10:43] <snack_> and I'd settle for a few days. a weekend. anything. :P
[10:45] <snack_> no issues as of yet... so I don't think we broke anything
[10:47] <snack_> quick question... does the amqp protocol allow for introspecting the queues on the broker? I'm currently doing this through the REST interface, which seems a little kludgy to me
[10:48] <antares_working> snack_: no
[10:48] <antares_working> snack_: and queue introspection in general is one of RabbitMQ's weak points right now if you ask me
[10:49] <antares_working> snack_: amqp gem 0.8.3 is out, bumps minimum dependency versions
[10:51] <kke> would it be difficult to duplicate beanstalk's TTR functionality as a rabbit plugin?
[10:51] <antares_working> kke: can you explain how it works?
[10:51] <snack_> antares_working: how about this: http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/11877
[10:51] <snack_> antares_working: are you going to do the default consumer thing?
[10:52] <antares_working> snack_: yes, absolutely. Otherwise I will anger RabbitMQ architect(s) :)
[10:52] <antares_working> this weekend
[10:52] <snack_> hehe
[10:52] <snack_> cool. you know you have a tester here
[10:52] <kke> antares_working: like queue.publish("poop", :ttr => 300) and that means if the message is received but no ack (or touch) for 300 seconds, it will return to the queue like it does in rabbit if the consumer dies before acking
[10:54] <snack_> antares_working: lunch, bbl
[10:54] <snack_> antares_working: thanks so much
[10:54] <kke> and have some counter, like meta.timeouts which would increment if this job has been returned the queue because it wasn't acked in time
[10:54] <antares_working> snack_: bon appétit
[10:55] <kke> +to
[10:55] <antares_working> kke: ok. This would require your messages to have some kind of identifier that does not change over time.
[10:55] <antares_working> kke: also, clustering may make this harder than I think but it sounds doable
[10:56] <antares_working> kke: also, Queue#publish is deprecated. Please do not use it. http://bit.ly/amqp-gem-docs Migration guide mentions why :)
[10:56] <antares_working> same goes for bunny, but it lacks docs :(
[10:58] <kke> that's pretty much the only thing from my point of view that beanstalk offers but rabbit doesn't without some complicated broker thing on top of it (celery? cloudist?)
[10:59] <kke> and what beanstalk lacks is replication
[11:00] <antares_working> kke: I agree that there is a few things rabbitmq could offer to make it a killer messaging solution. Ask rabbitmq team on the mailing list, maybe they will consider implementing something like this.
[11:23] <bob2351> rabbit 2.7.0 has been released
[11:39] <antares_working> bob235: should we expect new Java client build artifacts soon? I am not entirely sure 2.6.x ones made it to maven central at all last time :)
[11:50] <bob2351> the java clients will definitely be uploaded. when it appears will depend on sonatype/nexus
[11:55] <antares_working> bob235: thank you
[17:55] <Liquid-Silence> anyone here have experience with rabbit mq and C#
[18:13] <bob2351> Liquid-Silence: yes
[18:21] <Fitzsimmons> I feel like I might be misunderstanding how amqp works; is it feasible for one client to put some items on a queue, and and some point later another client launches, discovers the queue by whatever means, and the consumes the items there?
[18:23] <antares_> Fitzsimmons: http://bit.ly/amqp-model-explained
[18:23] <bob2351> Fitzsimmons: yes. you described a buffering scenario. amqp does that very well
[18:23] <antares_> Fitzsimmons: you can do that but very often AMQP consumers are long-running processes that have messages pushed to them by RabbitMQ
[18:23] <antares_> as opposed to polling
[18:24] <Fitzsimmons> that is a relief, because I thought so - but I'm struggling getting even a hello-world style setup to duplicate that scenario
[18:24] <antares_> and messages are not "put on the queue". Messages are published to exchanges and then routed to queues.
[18:25] <antares_> Fitzsimmons: maybe rabbitmq.com/tutorials/ or http://bit.ly/amqp-gem-docs will help. There is no shortage of "hello, world"-like consumers and producers for just about any client.
[18:25] <antares_> sorry, http://www.rabbitmq.com/getstarted.html
[18:25] <Fitzsimmons> I've tried taking an existing helloworld and splitting it up into two programs to help make the buffering scenario more obvious
[18:28] <Fitzsimmons> but okay at least I didn't make some bad assumptions, I'll keep trying
[18:40] <killfill> hi
[18:41] <killfill> rabbitmq beam.smp 65139 18 tcp4 *:55672 <--- i can see rabbit is listening to port 55672, but
[18:42] <killfill> cannot access de web interface
[18:45] <stageit> Hi, I have a question about scaling using multiple exchanges vs. one exchange and multiple topics
[18:45] <killfill> aah nm.
[18:45] <killfill> :P
[18:46] <stageit> What will scale to tens or hundreds of thousands of users better? Having one exchange that routes to one or two dozen topics, or just having one or two dozen exchanges that will have less messages coming in?
[18:48] <killfill> stageit: i know very little... but have the impression each exchange is an erlang process.. so i would have to bet, i would bet to have multiple exchanges.. :P
[18:48] <killfill> so if i have to bet i mean.. :P
[18:49] <stageit> Cool, thanks for the info. I guess we'll probably try both ways and do some benchmarks
[18:51] <Fitzsimmons> when I do a rabbitmqctl list_queues, is the second column the number of items in the queue?
[19:37] <stageit> Has anyone used the amqp ruby gem? It seems that since it is based on event machine it is impossible to open a variable length persistent connection (just by looking at the examples). What I'd like to do is open a connection so I can call various methods that will publish to it, and then close it at some point in the future. Is this possible?
[19:38] <stageit> Or, alternatively, is there a better gem or library to use?
[19:39] <Fitzsimmons> I'm struggling with that gem right now - I'm not really sure what you mean by variable length
[19:43] <stageit> I just mean that all the examples set a timeout for when the connection closes in the same method that they open it in. Id like to have a method that opens the connection and leaves it open, then publish to it some number of times and then close it whenever I'm ready
[19:43] <stageit> publishing and closing from other methods, that is
[19:49] <Fitzsimmons> from what I can tell from the examples, you can do something like connection.close { EventMachine.stop } if some_quit_condition
[19:49] <Fitzsimmons> I don't know precisely what that does since I'm just getting started, though
[20:31] <fryguy> if I want to use gevent to do asynchronous consumption of messages, what is the best python library to use? I was using puka but it's missing some features (notably basic_reject with requeuing). haigha has it's own event loop that doesn't seem to work well with gevent. Should I just come back to pika?
[20:35] <gmr> I'm partial to it
[20:36] <gmr> you want to run on a gevent loop?
[20:36] <fryguy> well, i'm not sure if I'm taking the right approach or not, but I want consume to dispatch a greenlet to execute asynchronously
[20:37] <gmr> ah
[20:37] <gmr> I've not used gevent so I'm not the best person to comment, though I'm happy to take a look at adding support. I expect to be working on the 0.9 tree again in the very near future.
[20:37] <fryguy> the consume callback is doing a bunch of thrift stuff that is very I/O bound, so I'm assuming I can get better throughput if I run those in greenlets
[20:37] <gmr> committed to have a few people to have 0.9.6 released before the end of the month
[20:39] <gmr> https://github.com/pika/pika/issues/102 for tracking purposes.
[20:57] <fryguy> is there any other way I can use an event_loop to do async consumes?
[21:24] <gmr> fryguy: pika supports 3 different event loops
[21:25] <gmr> the tornado event loop, a select based event loop I wrote (which is the slowest of the three) and a asyncore event loop
[21:26] <fryguy> but i'm bound by the execution time of the consume function, so I don't see how to make multiple consumes happen in parallel
[21:27] <gmr> oh I see what you're asking. I the consumer framework we use, I scale out with multi-processing with a connection per process.
[21:27] <gmr> I've see people use multiprocessing and queues to send data back and forth across processes
[21:28] <gmr> that is the multiprocessing queue primatives not the threading queues
[21:28] <fryguy> does that defeat the purpose of an event loop though?
[21:29] <fryguy> since we essentially have 1 thread or 1 process per connection now?
[21:31] <fryguy> and all of the associated memory, cpu, and context switching overhead?
[21:35] <gmr> well your issue is your processing of messages in your consumer is your blocking event
[21:35] <gmr> if you moved all of that to be asynchronous then you'd solve your issue
[21:36] <gmr> if you use one of the standard ioloops, you could use async clients for io related activities, if such exist.
[21:38] <gmr> so, using multiprocessing, you could have a thrift communication process that your consumer does nothing but put put items in a queue for it to send to your cassandra server (or whatever) http://docs.python.org/library/multiprocessing.html#pipes-and-queues
[21:38] <gmr> that would offload the consumer process to run at a higher transaction rate if you dont need guaranteed writes
[21:39] <gmr> if you can find an async thrift driver, you could do the whole thing with callback driven events and allow for multiple, simultanious messages being processed, but again, under the assumption you dont want to have a synchronous commit workflow where you ensure a message is properly dealt with before letting rabbitmq know it's ok to move on.
[21:41] <fryguy> i have a synchronous commit workflow
[21:42] <fryguy> well, not synchronous, but I need to either ack or reject a message based on the thrift response
[21:43] <gmr> so you're constricted to that timing regardless. If you're not using auto-ack, RabbitMQ should be waiting to give you your next message until you ack or nack
[21:43] <gmr> so you address that with parallel use
[21:43] <gmr> you could do this by binding multiple consumers, in a pure async, non greenlet driven method by issuing basic consume a few times
[21:43] <gmr> and finding where your painpoint is
[21:44] <gmr> that is at what point is your blocking io for thrift going to block other messages from being consumed
[21:44] <fryguy> hrmm
[21:44] <gmr> if, for example, the thrift driver is not async, then it's likely going to block on read
[21:44] <gmr> and then you're screwed.
[21:44] <fryguy> so if i issue basic_consume multiple times, it will consume multiple messages?
[21:44] <gmr> but if it's async, you could issue multiple consumes.
[21:45] <gmr> yes, each consume is independent from the other.
[21:45] <gmr> you could, for example, just to be clean
[21:45] <fryguy> the thrift lib is synchronous I think, which is why i was using greenlets to hide that
[21:45] <gmr> create a channel and consumer on that channel, then create the next.
[21:45] <gmr> i've not used greenlets to any extent beyond hello world, I dont know how that would impact pika.
[21:53] <fryguy> if i consume a message on one channel, can I ack it on another?
[21:58] <nedbat> can someone help me log or trace what's happening to my messages?
[22:15] <gmr> no, you'd ack ont he channel it came from
[22:15] <gmr> nedbat: happy to give pointers if you can give some info as to what you're seeing
[22:16] <nedbat> gmr, I think I'm on the trail of strange settings differences. But tell me, is there a way to just set log_level=DEBUG or something?
[22:17] <gmr> in rabbitmq?
[22:17] <gmr> hmm have not run across that
[22:18] <gmr> there is the firehose plugin which I've heard is good for trying to trace what's happening to messages
[22:18] <gmr> you can look in /var/log/rabbitmq/rabbit@[your-node].log and /var/log/rabbitmq/rabbit@[your-node]-sasl.log
[22:18] <gmr> that might give you some info
[22:20] <nedbat> gmr: i saw the firehose plugin, but if I'm a bit confused about rabbitmq and exchanges, etc, and where my messages are going, how will i somehow consume those new messages?
[22:20] <nedbat> i'm sure it's great for up-and-running systems, but it won't help me get off the ground.
[22:22] <gmr> gotcha
[22:22] <gmr> you're probably best to read a tutorial on rabbitmq before getting too deep
[22:22] <gmr> http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ is good
[22:22] <gmr> exchanges are routing mechanisms, in essense, queues are queues.
[22:23] <gmr> you have to bind queues to exchanges using routing keys
[22:23] <gmr> and you publish to exchanges+routing keys, not to queues
[22:23] <gmr> fanout exchanges ignore routing keys, topic exchanges allow you to wildcard routing keys and direct exchanges allow for just direct routing key to queue assignment
[22:25] <gmr> what language are you trying to write publishers/consumers in?
[22:27] <nedbat> I'm using Django and celery, and have them configured to use bindings to get to the proper queues.
[22:27] <nedbat> I guess I'm trying to understand why others don't want some kind of verbose logging to debug problems?
[22:30] <gmr> here's a presentation I gave in October that may or may not help
[22:30] <gmr> http://www.scribd.com/doc/72195631/Scaling-RabbitMQ-to-11
[22:32] <gmr> I'd recommend writing your own publisher and consumer to get the concepts 100% down before moving on to celery: http://www.rabbitmq.com/getstarted.html
[22:33] <gmr> to me, debug logging would require rabbitmq internals knowledge to make full use of
[22:33] <gmr> as far as celery is concerned, there may be some thing there for that, but it's a bit abstracted from amqp/rabbitmq
[22:37] <nedbat> rabbit internals? Somewhere isn't there something that could print a line like, "Got message <..>, routing to blah blah?"

These logs were automatically created by MyxoRoboto on irc.freenode.net using the Java IRC LogBot.