*** now talking in ##level3 *** topic is Hi all. Tracking the level3 problems. || http://scoreboard.keynote.com/scoreboard/Main.aspx?Login=Y&Username=public&Password=public verio is having some problems to at&t, cogent, internap, sbc level3 is stil showing very high latencies to everything but, well, level3 8) ok, it appears that the links went down, came back up and went down again, right now I'm hearing about l3 depeering off and on with cox and sbc. the global routing table is currently missing almost 2000 routes. since I seem to have received ops and lilo is probably overloaded: poke me if you have something to say the channel should hear thanks to everybody who provided a screen shot, I'm back online with scoreboard based on what I'm seeing it looks like level 3 and vario have their own private intranet, I am personally noticing latency on abovenet off and on, most likely from fallout. a lot of people are /msg-ing that not only level3 but also verio is having problems philips: if you have anything just sing out marienz: yeah, that's what I'm seeing marienz, confirmed I'm told APIC is still having problems ok based on what I just heard from a routing admin, level 3's routes are flapping quite badly Dampinfo: penalty 3022, flapped 10 times in 00:29:32, reuse in 00:08:19 also, not only lilo seems to have problems getting to scoreboard. Someone with the bandwidth to host a screenshot or mirror of sorts would be appreciated :) marienz, I can get there fine I'm also told there are high latencies to freenode, probably to several of our US servers verio is now down across the board if I had to guess I'd say they're reupping a lot of key routers lilo, watch xo let's see if it persists I think it might be next to go scoreboard is dying on me that's not a good sine erm sign 8) [|404notfound] i.e. http://scoreboard.keynote.com.nyud.net:8090/scoreboard/Main.aspx?Login=Y&Username=public&Password=public (for those who can't get at the scoreboard) someone just asked if this could be anything to do with EU wanting changes in the way root servers are controlled, the business with icann, eu and the us government it seems like a "dumb question" but it's reasonable if you don't know how this stuff works no, those issues are all at policy level also, msw provided http://www.rpath.com/~msw/internetpulse.png and MtDew provided http://sha1.net/images/scoreboard.jpg, but I don't know how much bandwidth they have. lilo, some of them are technical, if l3's flapping routes network wide, they'll simply get shut down. chances are nobody would interrupt service due to a policy dispute, not without a lot of public discussion so it's unrelated marienz: *nod* lilo, I disagree, some contracts come complete with NDA :P trelane: but any "public stunt" would be accompanied by publicity trelane: this almost has to be a pure technical problem lilo, l3 pulled it two weeks ago, I would be unsuprised to see it again, especially with the global information summit thing coming up. verio is still all down, level3 is getting weird stats down on some lines and up on others xo to level3 is down cox and l3 may be back up just got a report that verio to level 3 is down (i work for XO Communications) on the odd chance anyone here is one of our customers, we're clean to sbc, cogent, at&t, sprint, and wiltel most of the peering issues are due to Level3's $30,000 extortion (if I don't respond to a /msg I'm not ignoring you, I'm just slow/swamped. Sorry) guys it looks like it's a policy issue. * trelane wonders if l3 is entirely solvent if you need something through you can /msg me as well trelane: let's not jump to conclusions savvis, SBC, sprint, uunet, verio and XO have lost their link to level3\ ahhhh okay level 3 appears down to everything based on scoreboard well, regardless, technical or policy, it's certainly going to be interesting lilo, this could get REALLY ugly incoming from level 3 is almost all down, outgoing is spotty " [MtDew] I'm going to update mine every few minutes, I don't seem to be getting swamped too badly." that's http://sha1.net/images/scoreboard.jpg. If you can't access that, poke me, there are some more screenshots in my buffer. scoreboard is having problems, I'm sure it's due to load [sorphin] if you happen to be an XO customer, right now it's a clean route to irc.freenode via cogent due to Level3's issues (relay that please) Level3 is reading N/A across the outgoing board lilo: I see it perfectly here lilo: might be your routing to it :P UdontKnow: it's an error message from the site UdontKnow: not sure what's going on [prasys] just would like to inform you , APIC link to US Networks seems to be laggy again , but again we're still connected to it. It was okay just now and its starting back again. scoreboard is now reporting under maintainance marienz, relaying to all channels now (1129877486 04:51) tell lilo i've setup 15 secdonds refresh screept to get the html see the output at http://siberia.homeunix.org/level3/graph.html routing table is starting to reporpigate based on reports from an admin at an internet news service anyone have stats on the asia and european links? [wizardjames] http://wizardjames.robdavy.com/pictures/stats.png < around 20 secs befor it went down level3 is accepting some incoming with massive latentcy but is still N/A on outgoing Jtkiefer, routes still flapping if anyone has stats on the European, Asian and Oceanic linkups please /msg one of us [prasys] http://www.internettrafficreport.com/asia.htm <-- is seriously facing a lot of downtime , the avg current respond time is 711 ms ( level3 is going 0% on my view if you recently /msg'd me something that's still relevant, please re-send. (1129877587 04:53) <*Juggie> scoreboard now shows N/A for L3 for all routes [msw] http://www.rpath.com/~msw/internetpulse.png is up-to-date too iim seeing about 120ms from the us to the uk level3 started out as a high latency destination and it's now high latency both origin and destination lilo, it's a shitstorm I'm still getting a live linkup with the scoreboard lilo, the level 3 quarterly conference call was yesterday 10/21, MASSIVE losses http://finance.yahoo.com/q?s=LVLT&d=t [IceKarma] may be geographic, but I'm getting NXDOMAIN for internettrafficreport.com Savvis and SBC still mid range personally can't get to internettrafficreport.com either. y. Broomfield-based Level 3 (NASDAQ: LVLT - News) reported a net loss of $204 million, or minus 29 cents a share, on revenue of $799 million. trelane: well, if they want to go bankrupt, its their chance got a scoreboard mirror for anyone who can't get to the main one http://scoreboard-mirror.corelands.com/Scoreboard/Main.aspx?Login=Y&Username=public&Password=public looks like XO's about fixed any word on Verio? or on Level3? level3 is toast trelane: its only bad to intermap, verio and l3 I'm not so sure what verio's issue is trelane: others are 100% on xo my vario contact is in bed Udontknow, what's up with internap it's getting bad fallout They're giving out master ticket #'s of 1429209 1429184 and 1429189 --- Rumor has it a maintenance in Chicago backfired - from NANOG post checking nanog myself Jtkiefer: xo -> internap wasnt so good here Savvis, SBC, and Internap are all but down and out contacting each other Some of APIC links are down (even the site itself is down) , my ISP says that they using "backup routers lilo, what's +z? anyone who wants to send something to someone with ops can do so Jtkiefer: ops can see moderated you just talk into the channel we may not leave it that way but we'll try it (1129878053 05:00) <*sorphin> I'm w/ XO, we've been rerouting because of level3, we're clean through cogent on a quick check lilo, well people are still welcome to send a msg to me if they'd like Hosts which are inside level3 networks are getting routed through other networks if available (NYC Level3 facility which I have secondary access to) internap noc is requesting traceroutes of stuff that's down to and from them (1129878185 05:03) < *sorphin> 64ms from XO to freenode via cogent lilo, lets get administration at various noc's voiced http://wizardjames.robdavy.com/pictures/stats.png, a little over 40 secs old trelane: if we can get them in here I'm happy to trelane: /msg scrumptious hi HEre - people can see the state of l3's BGP accouncements into the global table from a Toronto.on.ca perspective at this URL: http://ops.rogerstelecom.net/cgi-bin/lg.cgi?router=SprintTOR&query=bgp&addr=4.4.4.4 lilo, I'm 70ms from www.internap.com on abovenet/chicago OrgulloKmoore: only ops will see you now lilo, can we get a voice on Myquil please, he keeps bringing in good stuff OrgulloKmoore: we're running on opmoderate mode if you're in a NOC for a major provider, just type 'PROVIDER' into the channel, all caps we'll try to sort you out *** UdontKnow changes topic to 'Hi all. Tracking the level3 problems. || http://scoreboard.keynote.com/scoreboard/Main.aspx?Login=Y&Username=public&Password=public | Msg an Op or a voiced user if you have information regarding the problems | we're running with opmoderate. talk on channel but ONLY ops will see it' thanks np just a reminder: < MtDew> http://sha1.net/images/scoreboard.png about 30 seconds old we're also clean through teleglobe to freenode Router entries from Toronto d 131ms http://ops.rogerstelecom.net/cgi-bin/lg.cgi?router=SprintTOR&query=bgp&addr=4.4.4.4 also, still feel free to /msg me if you have something important to share and we miss you in the fray here < msw> -- http://www.rpath.com/~msw/internetpulse.png -- just updated or any other voiced user level3-> others improving level3 and Verio are dead level3 has been bypassed, last check, we're dropping through TDS networks uunet appears to have in and outbound into level3 ... albeit intermittent with very high latencies SBC to internap is laggy XO to XO is laggy uh no it's not i'm sitting in our core routers atm sorphin, you a level3 person? no XO XO is 6ms to SBC sorphin, any idea what the problem with verio is, they've been hit especially hard by the level3 outage wildcard0 phoned level3, their comment is that it's a technical issue lilo: uh huh APIC link seems to be getting from bad to worst. For now , my ping to freenode is around 600-700ms with 5% packet loss and we know level3 doesn't lie ;) pretty nasty "technical issue" Jtkiefer: hold checking verio "< ohsix> http://www.level3.com/LookingGlass/", but I personally can't view this. global routing table is now missing only about 1000 routes trelane, they rebuilding the routing tables? Jtkiefer, appears so I can see the looking glass :) Jtkiefer: iirc, verio does extensive peering w/ level3, vs anyone else, we used to peer w/ them more than we do now (level3) realize that if bgp flaps too many times they'll get choked Level3 down in and out, Verio down incoming http://www.merit.edu/mail.archives/nanog/msg13106.html ok, correction ... just now i *could* see the looking glass :( well congrats to level 3, they crashed the itnernet internet i mean verio and level3 destinations are all high lag sbc to internap was completely down from 11:04 to 11:15 at which point it came back up and stopped dropping 100% packets at least level3 origin is high lag SBC and itnernap seem to be back up to normal * sorphin waits on freenode to stop trying to resolve my box at XO lilo, they will be until this starts to settle lilo, the problem is world+dog have suppressed level3's announcements summary for those who just joined: verio and level3 are having significant issues, and other providers are beginning to route around the problem. reports of some packet loss to level3 but some packets are getting through unconfirmed comment that comcast is considering charging level3 in turn (1129878659 05:10) <*EfentharWireless> cannot get to keynote through sprint now could you repost this for me? updated every 20 seconds: http://wipproject.zapto.org/tmp/scoreboard/mirror.html Baloogan: a backbone provider lilo, I dont think there will be a l3 to charge trelane: well, let's just observe the proceedings we were gonna do the same to level3 but we still ride some of their fiber d'oh nick125 got through to level3's LookingGlass and got http://pastebin.com/400674 <[sorphin]> ok < Jameno123> level3's internal routing tables are full again, outbound routes from their gateway faciliates are finnally allowing outbound packets ;) I'm also told it's possible comcast may file a lawsuit again, unconfirmed <[sorphin]> XO may file one, but we'll see <[sorphin]> up to the peering dept and legal < poningru_sleep> I can see the looking glass marienz, so we're getting outbound from level3 <[sorphin]> we did pay the $30k fyi dpower: no reason for them in particular <[sorphin]> shows $ buys notihng dpower: the person who mentioned it says he works for them incoming to level3 is still fucked up i can see the looking glass aswell, although I sometimes get a blank page in the middle thanks UdK lookingglass is http://www.level3.com/LookingGlass/ which from what I can gather allows you to check what level3 can and cannot access. I can't access this page myself, but apparently some people can. (1129878911 05:15) <*EfentharWireless> access to scoreboard through sprint has returned < OnyxRaven> NYC at least through Dallas is operational from a number of different hosts (cogent, sprint, qwest) I'm getting the lookingglass site *** lilo changes topic to 'Hi all. Tracking the level3 problems. || http://scoreboard.keynote.com/scoreboard/Main.aspx?Login=Y&Username=public&Password=public | Msg an Op or a voiced user if you have information regarding the problems | we're running with opmoderate. talk on channel but ONLY ops will see it || http://www.level3.com/LookingGlass/' do we know if connections are getting from coast to coast? scoreboard shows them coming back and how international connections are running? not getting any content from it yet though just framing I have some content from LookingGlass Quest to L3, Verio to L3, and Wiltel to L3 are still laggy if someone can screenshot or pastebin lookingglass output some people here would be interested * lilo nods at marienz marienz: can't access it anymore [My2Cents] http://rafb.net/paste/results/wujLb588.html marienz, do they know how to read bgp output? cant access the scoreboard anymore from here trelane: I have no idea. I don't :) <[sorphin]> brb ISP ratings http://sd.ihr.daze.net/ marienz, then lookingglass isn't too useful [03:18] connections seem to be traveling sprint (from internap) through level3 dallas to nyc on my end (asking about coast to coast) AOL appears to be having issues <[sorphin]> cleant rough WCG (Wiltel) to freenode <[sorphin]> clean even related note: http://www.thestreet.com/_excite/tech/hardware/10248488.html?cf=WSIWON1111051500&alias=/ht/nw if you can access lookingglass but do not know how to use it, or cannot access looking glass but would know how to use it, poke me so we can set up some exchange of sorts :) These updates brought to you by Level3 communications, your communications company :) <[sorphin]> 121ms through opentransit to freenode via XO getting reports aol.com is down <[sorphin]> my updates are courtesy of my being on our backbone :) marienz, if they're out atdn may be out which would take timewarner and rr.com <[sorphin]> trel1023: and charter <[sorphin]> grr lilo, any eta on when freenode is going to desplit ? <[sorphin]> trelane: and charter Qwest DSL users may have issues reaching internet sites that require traversing l3 ip space (thanks to wshs) I have some people who can access looking glass. If you want to ask them to run a query, poke me. aol.com seems to be down for some, up for others. <[sorphin]> any SBC customers shouldn't touch level3 at all to get here (i'm one, so i've done some checking) *** lilo changes topic to 'Hi all. Tracking the level3 problems. || http://scoreboard.keynote.com/scoreboard/Main.aspx?Login=Y&Username=public&Password=public | Msg an Op or a voiced user if you have information regarding the problems | we're running with opmoderate. talk on channel but ONLY ops will see it || http://www.level3.com/LookingGlass/ || For up to date info: http://www.merit.edu/mail.archives/nanog/' stats sites listings if people want to crawl these for info http://www.cybergeography.org/statistics.html <[sorphin]> 77ms through wiltel to freenode lilo, coverage on NANOG is still VERY spotty, though I'm not entirely sure my e-mail is trustable right now with the outages *** lilo changes topic to 'Hi all. Tracking the level3 problems. || DOWN: http://scoreboard.keynote.com/scoreboard/Main.aspx?Login=Y&Username=public&Password=public | Msg an Op or a voiced user if you have information regarding the problems | we're running with opmoderate. talk on channel but ONLY ops will see it || http://www.level3.com/LookingGlass/ || For up to date info: http://www.merit.edu/mail.archives/nanog/' reports of AOL down on west coast ok I have a confirmation that some parts of the internet are unable to reach AOL and vice versa ongoing reports of problems with AOL from various routes I'm sure they're just hurting along with everybody else I'm sure they're just hurting along with everybody else *** lilo changes topic to 'Hi all. Tracking the level3 problems. || Intermittent: http://scoreboard.keynote.com/scoreboard/Main.aspx?Login=Y&Username=public&Password=public | Msg an Op or a voiced user if you have information regarding the problems | we're running with opmoderate. talk on channel but ONLY ops will see it || http://www.level3.com/LookingGlass/ || For up to date info: http://www.merit.edu/mail.archives/nanog/' scoreboard and aol seem to be up for some, down for others now. lilo, when bgp gets this broke it does take awhile to piece it back together report that level3.com is accessible from alter.net again <[sorphin]> atdn and sprint are still happily talking and a report that it's not :) lilo i can confirm that, although from my side its a bit spotty with high lag of 5+ seconds problems at atlanta's level3 router from alternet I have LookingGlass output, one moment West coast appears to be bypassing level3 in some locations can we get a confirmation on that? bear in mind, folks, that almost any reports that "foo is down" are actually problems between two backbone providers at this point level3 and somebody, or maybe verio and somebody I have a report SBC switched to Sprintlink in the Sac and Bay areas l3 is lacking null-routes, while trying to access their looking glass, I can see a few routers bouncing packets till they TTL. lilo, correct good news , APIC links seems to be up , at least I can acesss their website and lag spikes to freenode are dropping also, bear in mind that due to router problems it looks like a lot may be down for some, up for others. Probably not particularly interesting if a particular site does or does not work. MyQuiL, cute. also bear in mind that though things may be technically fixed due to flapped routes and suppression it could be up to an hour before we see the effects of changes that were made still running about 900 routes below normal I don't know the quality of reporting, but I have a backgrounder on the Cogent/Level3 interruption a couple of weeks back: http://news.com.com/Network+feud+leads+to+Net+blackout/2100-1038_3-5889592.html Judging by what I see on NANOG about guys 'round the world losing their BGP feeds completely, I'd envision this as something like "how to setup an Tier 1 network in 15 minutes or less." This is something you'll be reading about in the papers. Verio is down ingoing, Level 3 is down outgoing and ingoing this appears to be financial issues with level3 resulting in deliberate peering interruptions MyQuiL, maybe on CNN "BREAKING NEWS" heh <[sorphin]> lilo: indeed <[sorphin]> as i've said MyQuiL, agreed, you know, they say it should survive a nuclear attack, I'd say this is an example that it DOES WORK. <[sorphin]> extortion they've spent a lot of money on infrastructure and are trying to get payments from peers lilo, right I think I mentioned that <[sorphin]> funny can someone put a call into CNN in Atlanta <[sorphin]> we even paid well luckily the internet is designed to deal with damage like this trelane: partially this is the problem with over-centralization of networks and insufficient redundancies. I'm not noticing a big drop of traffic on my hosted sites here tho. [sorphin], you don't just pull the plug, you know that, you have to have something to blame hte "outage" on MyQuiL, it doesnt have to be instant alot of good the upgrades have done them! ... this just shows, no matter how much redundancy a major ISP has, they can still go down and take half the internet with them if a huge chunk of netowrk falls off and is replaced in 6 hours that's pretty damn good <[sorphin]> trelane: yeah, they blame it on people not paying $30,000 WildPikachu, concur [sorphin], bingo anyone else showing weird stats from XO to XO and from Quest to level 3? <[sorphin]> Jtkiefer: XO to XO is fine <[sorphin]> 6ms or less iKernel: we can see it hmm iKernel: normal users cant when things are back to normal, people should google 'BGPlay' and plug in a level3 prefix (4.0.0.0/8) and watch what happened around these times. UdontKnow: levelone please sorphin, scoreboard is reporting 92ms latency <[sorphin]> fyi on anyone who doesn't know <[sorphin]> Jtkiefer: scoreboard isn't sitting on our core routers <[sorphin]> I am editorial comment....as usual, GNAA have picked the opportunity to run botnets.... it just amazes me that people can be trivial enough to cause distractions when folks have actual significant problems on their plates ;) it's the getting-a-life thing hey, gotta keep yourself occupied when your favourite pron sites are inaccessable no link to L3 Berlin through Sprint update for those who are just joining us, level3 FUBARed and the internet has performed a major netsplit i just lost uunet to level3 again, access to scoreboard is intermittent, they seem to be having server problems.... my guess is out-of-memory due to load I can't make heads or tails of this output myself, but for those who can: < antarus> http://pastebin.com/400681 -> BGP output from my area ( Merit ) to the rest of the country <[sorphin]> Jtkiefer: fyi, here's a small shot <[sorphin]> 3 p4-3-0.MAR1.MarylandHeights-MO.us.xo.net (207.88.84.73) 0.654 ms 0.660 ms 0.570 ms <[sorphin]> 4 p5-2-0-2.RAR1.Chicago-IL.us.xo.net (65.106.6.157) 6.647 ms 6.624 ms 6.706 ms <[sorphin]> 5 p0-0.IR1.Chicago2-IL.us.xo.net (65.106.6.134) 6.729 ms 6.603 ms 6.546 ms Jtkiefer: not often you see the whole internet split <[sorphin]> 6ms from st louis to chicago I'll have LookingGlass output soon 03:31] On October 21st we began receiving numerous calls from multiple location of routing issues. Upon further investigation we discovered that Level3 appears to be a common factor. We have discovered that Level3 is having major network issues nationwide. We are in the process of shutting down our Level3 peering at all our locations. We have opened ticket 184739 to track this issue. Currently Level3 does not have an ETR. We will u [03:31] noc@internap DJCapelis is loading it onto athena Links between sprint and L3 Brussells and Boston down shit what about UK to US northeast you need a cloak ??? nanog.org -> resources got many looking glasses if you guys are wishing to dig around those are the major cross atlantics Jtkiefer, waiting on results ok, what do I do? if those go down we lose the link to europe salut poulette An:P Anthere: no.... if wikinews/wikimedia people have any info, just pass it on here salut biche lilo: what kind fo info? <[sorphin]> Jtkiefer: we're clean to teleglobe delphine: anything relating to the level3 business Links between sprint and L3 Dallas and Detroit down hmmm, I fear most developers are already here... delphine: you guys seem to make it a point to collect information :) http://neptune.dti.ad.jp/ a few looking glasses here <[sorphin]> 130ms <[sorphin]> XO to teleglobe to freenode lilo its a network admins job ;) <[sorphin]> as of 10 secs ago Links between sprint and L3 Houston, down lilo, you logging this channel? more reports from our previous informant about router hand-offs in atlanta, packets going in circles houston level 3 routes appear to be back up Unixy folk might want to setup 'smokeping' to watch this sort of fun in the future. <[sorphin]> fyi, routing isn't necessarily the same bi directionally folks will probably be interested in this: http://athena.numist.net/~djcapelis/lookingglass.txt Links between sprint and L3 LA, Down okay, now this is interesting l3 on the 30-60% availability now to most others on scoreboard Jtkiefer, london to sprint is up No, routing is rarely the same bi-directionality, your ISP will choose which peer to use based on various agreements and path-lengths. Looking glasses are really the true way to trace. that's good, at least we're getting europe routes okay, this is interesting.... a level3 customer says they heard back from them, that they were performing some network-wide software upgrades and ran into problems <[sorphin]> also coast to coast U.S. seems to be having issues the tone of the comment suggests that this has happened before no word from asia for awhile Links between sprint and L3 LV, Manchester, down MyQuiL: see my previous link : level3's lookingglass Germany down and I mean 'looking glasses' where you say things like 'sh ip bgp $prefix' <[sorphin]> 121ms from XO to OpenTransit in europe to Freenode or so it seems let me check T-Mobile down for sure [sorphin]: have we never asked you guys for a server? :) down or unreachable * lilo makes a note <[sorphin]> haha numist: can't load it, but here's a LG in Toronto: http://ops.rogerstelecom.net/cgi-bin/lg.cgi?router=SprintTOR&query=bgp&addr=4.4.4.4 lilo, this is amazing that we can get all this info all at once <[sorphin]> lilo: backbone is OC-192 MyQuiL: want me to pastebin it? atl l3 routes appear to be fully up, as well as houston l3 datacenter inbound/outbound traffic, and houston l3 network traffic <[sorphin]> hell out of my building <[sorphin]> 2xOC-12 <[sorphin]> :P [sorphin]: well, I'm convinced :) gah, seamonkey went fubar Links between sprint and L3 NY, down numist: finally got it. nice set. :) Links between sprint and L3 Oakland, down not a small file, and athena is my home router :o * MyQuiL wants a BGP feed ... gunna have to talk to the NOC guys tomorrow. I hear that links between sprint and l3 (ny and oakland) are down anyone have info on the Asia links yes, oakland is down <~xbmodder_lappy> San jose is level3's connection to speakeasy thouse are west to east right? Jtkiefer: probably something on the lookingglass Links between sprint and L3 Phoenix, down numist, I don't know if it'll load for me <~_[iceman]> according to L3 Lookingglass, i have connectivity to L3 Atlanta, but 200Ms pings Jtkiefer: doesnt have to Verio is all down LookingGlass from my IP 64.36.4.146 in San Francisco to ALL SITES: http://pastebin.ca/26146 route count is now 600 below normal Jtkiefer, link from wikimedia korea cluster to florida is working fine uploading my looking glass output to a proper server links between telus/shaw and level3 washington, down http://numist.net/index.php/Level3 Links between sprint and L3 Seattle, down Jtkiefer: ^^ lilo, wow <[sorphin]> not that anyone cares Seems a traceroute from toronto is getting partway to level3.net and then looping forever within level3. <[sorphin]> but peering w/ Ebay and M$ is normal on XO's network :) Geniva and Hamburg -> freenode is active but lagged numist: d'you have a feed? or are you scripting that? or just putting it together by hand?