Kb252

Emulab FAQ: Testbed Operations: How do I deal with a non-existent experiment with allocated nodes?

There are nodes in the reserved table associated with a non-existent experiment:

  mysql> select node_id,pid,eid,rsrv_time,vname,erole from reserved where eid='team4';
  +---------+-----------+-------+----------------+-----------+-----------+
  | node_id | pid       | eid   | rsrv_time      | vname     | erole     |
  +---------+-----------+-------+----------------+-----------+-----------+
  | pcwf16  | CEN5540P2 | team4 | 20060321160016 | A4        | node      |
  | pcwf7   | CEN5540P2 | team4 | 20060321160016 | A1        | node      |
  | pc133   | CEN5540P2 | team4 | 20060321160017 | G4        | node      |
  | pc134   | CEN5540P2 | team4 | 20060321160017 | tbsdelay0 | delaynode |
  | pc144   | CEN5540P2 | team4 | 20060321160017 | G3        | node      |
  | pc147   | CEN5540P2 | team4 | 20060321160017 | G2        | node      |
  | pcwf8   | CEN5540P2 | team4 | 20060321160016 | FTP2      | node      |
  | pc131   | CEN5540P2 | team4 | 20060321160016 | fw        | node      |
  | pc275   | CEN5540P2 | team4 | 20060321160015 | G1        | node      |
  | pcwf1   | CEN5540P2 | team4 | 20060321160015 | A3        | node      |
  | pcwf5   | CEN5540P2 | team4 | 20060321160015 | FTP1      | node      |
  | pcwf14  | CEN5540P2 | team4 | 20060321160015 | A2        | node      |
  | pcwf15  | CEN5540P2 | team4 | 20060321160015 | V         | node      |
  | pc130   | CEN5540P2 | team4 | 20060321160016 | G5        | node      |
  | pcwf18  | CEN5540P2 | team4 | 20060321160017 | C         | node      |
  +---------+-----------+-------+----------------+-----------+-----------+
  15 rows in set (0.00 sec)

This was one of the firewall experiments that failed. Probably related to the wifi nodes. But I have not been able to track down how/when nodes got left in reserved.

I will just clear these "team4" rows from the reserved table and look around a bit more for other "team4" state, unless someone has a better idea.

The other places where state exists are: delays, v2pmap, vlans.

> Sounds like something really went bad during one the recent mysql
> lockups.

> > I'm guessing that I cannot free them up, that nfree will see that the
> > experiment they are in doesn't exist and will choke?  But I will try.
>
> Just remove the entries from the reserved table by hand.
>
> 	delete from reserved where pid='xxx' and eid='team4';

I created an experiment emulab-ops/team4 using an NS file that used the same vnames and tb-fix-node'ed all the right nodes.

For all those tables with info for CEN5540P2/team4, I changed the eid to emulab-ops. Now all the nodes were hooked up with my experiment.

Then I changed the experiment state to active and swapped it out. Everything seems to have been cleaned up as it should have.