Elevation motor counterbalance

After a suggestion from Eric, Brian came up with a counterbalance to the elevation motor. There is a spring now that is mounted to the bracket that holds the motor and the gear reducer, and is in turn connected to small friction wheel. The whole assembly is in the middle of getting painted, so try to ignore the gray primer.

The idea is to keep the two friction wheels aligned properly, and to prevent uneven wear on the wheels, something that we noticed on the BIMA wheel.

The hard part is to try to figure out how much compression should be on the spring. I ordered some heavy duty pressure indicating film to figure this out, and I intend to send it through the roller. Stay tuned for the results.

You can also see the rain guard that Brian rolled from Aluminum. It fits very well.

The perfect polarbear picture

Well, not really the best, but I wanted you to get a sense for why I think it is so hard to get the perfect polarbear telescope picture.

First- here are some reasons that now is a bad time to get a picture:
1. The sky is too clear.
2. The pad is a mess
3. I am the only one here, so there are no people to pose for scale.
4. I need some more equipment.
5. The elevation does not work again, so I cannot get the perfect angle.

Here are a series of 4 photos, all taken on the same night, all within about 2 hours, and all taken from the manlift. It is surprisingly difficult to hold a camera way up there.

This first photo is taken just before sunset, while the telescope is totally in sunlight. I don't like it because the sky is boring, the light is harsh, and the shadow on the trees is very dark. Taking a photo of a large white object also throws off the exposure.



This next one was taken as the Sun was below the trees. It is okay, but the horizon is all skewed.



This next one was with the sun even lower below the horizon. I think this is my favorite, and that is why it looks more processed. The sky is becoming more interesting, and I got he position of the manlift right. The color is not textbook, but I never like perfectly corrected color. One big problem though is that it is much darker than it looks, so the camera had to be pushed to higher sensitivity, and the sharpness is just marginal because of camera shake.




This last photo was taken just before the sunset. I left the white balance set for daylight, just so you can see how weird the color gets up there. The light was so low that none of the photos were usable due to camera shake. I want to try out a camera clamp to see if I can push the exposure longer.


We dont have long before we lose the manlift- just a few more weeks, then it will be much harder to get this shot. I am waiting for a night that is more interesting, with lots of clouds and almost raining.

some more critters

This family of odd birds was spotted at the high site. They are often seen avoiding your car on the way up to the site. I hear that they make good hunting. The are called Chuckers, and look like some variant of quail.



I have no idea what this is, but I left the door open on the container for a short time and he made it in. I hope it does not bite, but he was plenty angry when I trapped him in a plastic cup.

Servo Selloff

These last few days have been all about getting Track beam mode to work. Basically, it was Nate making and running a scan, Brian analyzing the data, and the Richardson guys changing something, then repeat. We were not able to fix the problem, but we have some ideas, so I will come back in a few weeks to try out the solution.

So we decided to do a conditional sell-off, one where we check out everything except for Track beam mode. Brian #3 came today to witness. We did turn out some minor errors that were fixed, and we did end up signing the document. Bryan #2 was the third witness.

Next week, I come back to finish up some loose things, and support Brian #1 while he paints the telescope.

I am way behind in posting photos, so here comes a random selection from the last couple of weeks.

We took this photo of Adrian about 1/2 hour after Track beam mode started working, essentially making Polarbear a whole telescope. We had about 5 minutes before sunset.


Nathan spent a lot of the time looking just like this. Staring intensely at the computer, spitting out lots of scan.


As you can imagine, the manlift is pretty fun. Both Adrian and Bryan now know how to drive the thing.



There was a crazy dust devil that hit the CARMA array, apparently, it happens a lot.



We sent Adrian into the big hole to do the last cutting of the tabs. The cable wrap currently works, but we will watch it closely.


I think that Adrian liked it in there. Dave and Daniel scoff because this hole is luxurious compared the the hole they had to sit in for 5 hours.


Brian #1 had to rescue Adrian, I think. Otherwise, I am not sure what he is doing.


Hans really got into priming the counterweights, but Brian #1 put us both to shame when he started using the correct roller.





And finally, the man of the hour, Radek and his family came for a visit. He was impressed.

Track beam mode

Barry, a software guy from Richardson, spent all Tuesday fixing track beam mode. It now works, and we've begun testing scans. We started with some fast circle scans:




The first tests showed 3 arcminute peak error in elevation and about 1 arcminute in azimuth. Sean remembered that the servos were set back to default for that, so he retuned the position loops back up to 0.5Hz and we repeated the test. Elevation and azimuth both came down to about 1 arcminute.

Next we started looking for excitation of resonances. On turnarounds, where velocity goes to zero, we often see resonances at 2-5Hz; the frequency doesn't seem stable. We also see some of the lines we saw in the accelerometer out at 20 and 40Hz. To simplify the problem we switched to purely azimuth or elevation scans, and saw a little bit of coupling between az and el - the servo which isn't being driven sees a wobble of about 0.01 arcmin peak.

At this point we noticed something really strange - a glitch in the encoders. For one sample, the rror jumped by about 1.5 arcminutes, which is mechanically unlikely. In the raw data coming back from the ACU, it looks like the old encoder value held over one sample. The ACU sequence ID, which counts the 100Hz clock which synchronizes all of our readout, increases correctly, so the software buffers are behaving. Later, in another chunk of data, we saw a big string of these glitches:
This plot is sinusoidal scan in elevation at fixed azimuth. From 50s to 150s, the drive appears to behave correctly: peak error of 1 arcminute and a small spike where velocity goes to zero, maybe from momentary static friction. At 150s, the behavior changes dramatically. The values from the encoder appears to occasionally fall behind for 1 sample and then catch up. After a few more seconds it falls behind by 1 sample for a duration of several samples. After 25s it's permanently locked into falling behind and contributes an error proportional to the elevation velocity.

Observation scan

20 degrees, 4 degrees / second, 2 degrees / second / second

Balancing the Telescope

Last week, we worked on balancing the telescope around the elevation axis. The telescope was heavy toward the boom and toward the primary mirror, so weight was added to the lower back. 250 and 400 lb steel plates were added. The torque required to move the elevation was reduced, and is likely now in an acceptable range. We need a better scale for our static testing

Servo tuning

After Al and Brian fixed the El drive alignment they handed Polarbear off to Sean for servo verification and tuning. Polarbear has 3 distinct servo loops. GCP sends position commands to a position loop, which sends velocity commands to a velocity loop, which sends current commands to a current loop. The current controller is factory autotuned once it's hooked up to the motors, but the velocity and position loops must be hand tuned.

To tune and verify each servo Sean repeated two basic tests: network analysis and impulse response, looking for the same things we do in bolometers - bandwidth, resonances, time constants, etc. He runs the tests largely by hand - turning the knob on a function generator over 30 seconds or so.

In their servo system, we have some margin to play. As long as we remain stable, we can trade between overshoot/settling time and bandwidth. Since we want to do fast scans with out needing much pointing precision, Sean cranked up the position loop bandwidth from .22Hz to .5Hz - the spec is .25Hz.


The velocity response of our telescope has a resonance at 3Hz due to elasticity in the drive train:

This sweep goes from .1Hz to about 15Hz. The measured velocity not coming up to command velocity on one side is due to a problem with the DAQ that acquired the data. At 3Hz the measured velocity phase slips about 90 degrees behind the command.

Elevation drive alignment

After adding 1000lb of counterweight we ran a balance test using the scale. We measure the force to hold the telescope still at several elevations. While driving the elevation between two positions someone noticed that the El drive was walking far enough off to drag the drive mount into the side of the primary friction wheel. This must have been happening for awhile, because the paint is scratched off all along that side.



The azimuth drive avoids this problem by holding the wheel in place with a pair of jack screws above and below the tensioner.



Al and Brian were able to adapt that to the El drive. In a day they had welded up new brackets and punched holes into the back of the boom to mount them. We've run the elevation quite a bit since then and the alignment is holding steady.


We went up onto one of the BIMA platforms while they were shutdown for generator maintenance to check out their drive, which ours is modeled after. Huan noticed that the two jack screws touch the same plate, so that they are very close together, while ours are fairly far apart. That constrains the wheel from turning, which could cause long term problems.

We're going to try pressure sensitive tape to check uniformity of the contact, then we'll see if the system needs improvement

BBQ

The CARMA observers invited us to barbecue Saturday night at the high site. it's quiet during the weekend so there were only 7 of us: Adrian, Huan, Nate, me, the 2 observers and a visiting friend of theirs. They have a gas grill, so the turn around time between working and cooking is fast.


Adrian periodically basted the ribs with barbecue sauce, and the observers had some fresh vegetables. Their friend is from Indiana, so he had very precise instructions for grilling corn on the cob.

Unlikely Visitor

I was at the telescope late one afternoon when I heard a big thud on the utility container. I was not expecting to see a seagull out here in the middle of the desert mountains. The last thing I want out here is a flock of seagulls, they are the rats of the air, but I do not think that we are in any danger of that.

Secondary Adjuster and cable wrap

All the parts for the secondary adjuster finally arrived, and Brian's contraption is done. It is hard to tell from this picture, but he replaced an unbalanced jack screw with this threaded rod that is not welded into the structure. There is also a 3/4 plate that is attached to the stainless box. Together, these form a push-pull mechanism that should make it easier to adjust the secondary.

I am afraid, however, to test it since we have photogrammetry for where the secondary is right now. I will have to come up with a way to measure it's current position before we tweak it.



Also, when I was slewing around the AZ, I found this other failure mode of the cable wrap. It turns out that the ends of the 'U' can touch each other, and sometimes when they do, the tabs on the outside catch. Al and Dave had previously cut off the ends on the inside, so when the 'U' collapses, the tabs no longer catch. I am wishing that Dave was here, so I cold send him into the hole to deal with this. It will probably take 1.5 days to get this done.

Air conditioner saga

I initially did not have an air conditioner on our container because the Japanese in Chile warned me from getting an american unit that cannot be serviced in Chile. I also hoped that we could 'get by' but this summer has proven to be a scorcher, and so began the air conditioning saga.


I started with this lousy 8000 BTU unit that I stuck in the vent opening. This thing was only psychologically effective, the temperature did not go down inside at all.


Unit #2 was supposed to be a 14,000 BTU portable unit. After the unit was late by 1 week, I started calling the shipper every 3 hours, and finally decided that Yellow Freight had lost the shipment somewhere in LA. Amazon did finally refund our money.

Unit #3 was an emergency acquisition from Kmart, another 8000 BTU unit, which worked only if you were in line-of-sight. The container is so air tight, that it also caused a serious vacuum in the container, so every time you opened the door, your ears would pop.


And finally, I just in stalled this a couple of days ago, what I should have done 6 months ago, a split, ductless, R410a system. It is actually not that expensive if you install it yourself. It is 24,000 BTU, runs on 208V 60Hz, and is surprisingly quiet, because the compressor is outside.

I installed it just in time for it to get fridgid here, so I haven't really been able to test it in 100 degree weather.

If I installed the refrigeration lines with a leak, then it is legal for me to top off the system with R410a, a non CFC based refrigerant. The only problem is that if we want to run this unit in Chile, we will need a 60-50 Hz frequency converter. I think we might already need one for the water chiller anyway.

New car

Brian traded in his awesome WRX for a brand new WRX STI. If there is any car made for this area, this is it.

I took this photo to make Daniel feel inadequate because he also has a non-STI Subaru, but really, it just makes me feel inadequate to be driving my POS rental.

counterweights and controller

It has been a week since the last post, so here are some of the things that went on.

First, Brian and I are currently the only people here. There are plenty of ethernet connections now.

Brian and I did a measurement of the imbalance with a heavy duty hydraulic scale. We where able to determine that there are 9000 ft-lbs of imbalance, with the CG toward the shields. It turns out that this measurement comes very close to the measurement from the motor current that Sean came up with. I was hoping for better. I think the reason we are off is because of the baffling, it was just heavier that I estimated.

This imbalance is bad for a couple of reasons. First, I talked to the designer of these drives, and he definitely thought that the friction wheel was not designed to hold that kind of imbalance. Secondly, if the system where to fail, it would end up with the boom down, so not very safe.

To fix the counterweight problem, Brian and I decided to go with Steel. This is after we scambled around looking for lead or other solutions. The OVRO staff told us to go with a welder in Bishop, and they will, source, cut, and deliver our counterweights for $2 a pound, which is not bad. The couterweights will show up on Tuesday, then we have to paint them, and then on Thursday we are having a boom truck come out here to install them. This should get us within a few hundred pounds of balance, and then we can add trim weights.

A second problem that developed has to do with the servo. I think that I burned a chip on the AZ motor controller. Mike sent me an accelerometer so that I could look for resconances in the structure. I was slewing the telescope 360 degrees in 90 seconds, then stopping and turning around. After looking at the accerometer data and trying again, I foind that the Baldor controller was complaing of a low DC bus voltage. These things are suppossed to protect themselves from events like this, so I am not sure how this happenned. After talking with Sean, he decided to get a new unit, and will bring it next week. Until then, we have to move the AZ the old fashioned way, by pushing.

Track beam problems

Sean came from Richardson TX about ten days ago to start testing out the servo. After some quick initial success, we began to find some problems. The first problem was a velocity limit in el, fixed by reprogramming the motor controllers. The next problem was an oscillation in the position loop in Azimuth. After a couple of days of back and forth with Richardson, they were able to figure out that it was some arcane software lock, which was failing to recognize a hard-wired rate limit. The az appears to now go to a position reliably.

The next step is to tune the parameters, but I am not sure we can do that until I am happy with the balance, more investigation is needed.

The current headache is that we cannot get track beam mode working. This has been anticipated, since SPT had some major issues getting it to work. We seem to make tiny bits of progress every day, but it is slow going because the system is so complex. I spent, for instace, some quality time on the floor of the contiainer fine-tuning some resistors so we could get exacly 10 microsecond pulses. Not fun, but it did appear that we got one baby step closer. The telescope is at least following the scans, and we saw for the first time a basline polarbear scan, 15 degrees back and forth.

We are still not getting any replies from the ACU, so event though the telescope is moving, we would not be able to tell where it is moving without looking at it.

Assembly almost done

The major assembly of the telescope is nearly finished. We have reached the six-week mark, and Albert has left for a week-long trip home. We hope to have everything done in two weeks.


Marius left for home a couple of weeks ago, and was replaced by Nathan, but here is an old photo of Marius painting the saddlebags. He and Dave took a lot of abuse for how they went from space-qualifies looking before the paint job to little rascals clubhouse looking after the paintjob, but the automotive paint will do the job, and it was a pretty smelly and messy job. I still think that Marius has some paint on him somewhere. The saddlebags are now installed, thanks to a very careful layout job from Brian and some help from Mike with the forklift. I am afraid that we will need a taller ladder to reach these saddlebags.



The baffles have been slowly going on for the last couple of weeks. The panels go on like a complicated 3D puzzle, where each piece had to be slightly trimmed to fit. "Tony" got a lot of grief for this job also.


We had one baffle casualty when a dust devil caught a sheet before it was fully attached. The sheet metal was somewhat mangled, but they banged it back into shape and I think it is okay for now. I will request a new one when more baffles are being made.



The last major piece went in a couple of days ago when we finally installed the cryo mass dummy. Al decided to install the frame first, then the ballast weight second. You can just barely see the Pullz-all that we used to hoist things into place. The manlift stood in for the eventual gantry that needs to be made here (80/20 Marius?).

Tony did not get any credit for how everythign fit together on the first try, because Tony did not fabricate these parts.


Here you can see the weights installed. It is a little scary to stand right below these weights, so I dont recommend it.


Here is how the telescope looks now.The look is basically how it will be for the next few months. A few more panels need to fabricated aroud the secondary, and an enclosure the the reciver will need to go on, but it wont look much different.


With the assembly winding down, I am trying to figure out the exit strategy. Of course, the major outstanding item is the servo, more on that later. Brian will be working refining the secondary adjuster of the next week, and I think that we will need substantially more counterweight, we seem to be heavy toward the lower boom, but I am not quite sure yet.

Fire and rainbows

This is still me playing catch up. There were a couple of days of crazy forrest fires, started by lightning. We could not weld those days. You could see the smoke from pretty far away, even from my Super-8. As it turns out, the fire was on Brian's front door, and he was evacuated. He toughed it out for two days in the campgrounds.



It has rained probably half of the days during the six weeks we have been here. The Cedar Flat site has only been operational for a few years, and they have never seen anything like it. They are blaming it on El Nino. We caught this impressive double rainbow over Polarbear late one afternoon.

Photo catch up

This is a catch up post showing some photos during the week of photogrammetry. It feels like ancient history now, so much has happened since then, but here goes.

Brian was in charge of placing most of the targets. Not too close, and every shot needs at least 3 'coded' targets to situate the photo. The rest of the spots are the actual targets. The first round also needs an "iron cross" to define the inital coordinates, and scale bars to define the scale. The process is somewhat magical, but if you think about it hard enough, it makes sense. The whole setup costs ~$200K and requires a significant amount of training to get it perfect, but they claim better that 0.001" accuracy.






There is a photogrammetry rhythm that is followed every night. It starts with making adjustments to the secondary, then waiting until the telescope thermalizes, then riding around in the manlift and taking abotu 100 shots over about 270 degress around the telescope. Here's a photo as the sun is sinking far below the horizon. Jay is messing with something, and Brian is adjusting some panels before we start.



We foolishly operated with the doors open most of the time, to keep things cool. Marius's fly traps have been redeemed, and there is a now a disgusting rogues gallery of bugs stuck to one of the traps. The liquid trap is still empty.


At least one night was actually clear enough to see starts. Most nights were cloudy and rainy.



This is an early photo of the targets. By the end, we had many more targets so I could check the warp in the panels.


I saw the sunrise on many occations. This will probably be the first out of hundreds, so get ready.