Cisco AnyConnect VPN (SSL VPN Software) was throwing “Failed to install and start agent. Installation has failed.” on me after logging in but before connecting (which was failing). I ended up randomly trying to go to the https site in firefox and it threw an error about the ssl certificate, some random number that was pretty meaningless.

This inspired me to go to ASDM, Configuration, Remote Access VPN, Advanced, SSL Settings and under certificates remove the certificate I was using for the interface. This problem went away! I have no idea what it didn’t like about the certificate. It would have been nice if it said something about this though. Seeing some errors about the client trying to access “https://ipaddr/CACHE/stc/1/VPNManifest.xml” is actually what got me playing around with the browser otherwise the only readable errors in the event log (the software makes it’s own section in the event viewer) is “WINDOWS_ERROR_CODE”. Nice. The debugging in ASDM wasn’t helping much either.

After that I was getting a bit further but failing and this time ASDM which was set to logging debugging was giving me a “TunnelGroup GroupPolicy User IP No address available for SVC connection” error. This turned out to be because I was using dhcp-server on the policy and pointing it at the broadcast address. This had worked for other cisco gear in the past. I watched the DORA process using wireshark on a server and saw that it was just repeating the discovers and offers over and over. When I changed dhcp-server to point at the unicast address for the server, the connection finally worked.

It’s taken me a couple weeks of staring at books and screens to figure this out. Please let me know if you think I’m bullshitting you.

Cisco Systems’ Simple Certificate Enrollment Protocol(SCEP) is for providing a protocol for Cisco’s routers, vpn concentrators (vpn 3000), access points (1130AG) and firewalls (asa 5500) to get the root certificate and get it’s own certificate “in band”. The “In band” means it does this over the network rather than you having to paste a BER x.509 certificate or some other “out of band” method.

It does not, as I was thinking for some reason, allow the client to obtain it’s own certificate from the CA by way of the device.

SCEP really isn’t needed for EAP, as the certificates are passed to the RADIUS server (IAS) which then does the required work (comparisons, validation, etc). PEAP (Protected EAP) supports using both certificates (and smart cards) and ‘secured’ passwords for authentication.

line protocol is down (monitoring).

duh. ‘show monitor’.

It seems like while Cisco always supports lots of security protocols, it’s a horrible world of acronyms… and I work in a horrible world of acronyms.

I did get PEAP going, finally. Prerequisites: IAS installed, with the remote access policy set to grant and ‘EAP Method’ on the Authentication tab of the Profile allowing PEAP. Best install MSCEP too.

Note that getting certificates and password working for amount to: ‘EAP Methods:’ set only the PEAP type. Edit that. On the EAP Types here, Add both ‘Smart Card or other certificate’ and ‘Secured Password’. I have Smart Card or other certificate first. Don’t worry much about ‘Fast Reconnect‘ unless you’re using multiple APs, in which I hope you have a better idea of what’s going on than me.

First, set the time. I couldn’t find ntp supoprt so use the ‘clock set’ command and set the time. After some debugging trying to get certificates working with MSCEP, I figured out that it wasn’t accepting the certificate because it thought it was 2002, and thus the certs weren’t valid yet. Make sure you set up the correct summer-time, heh. My time config as well, set in global configuration mode:

clock timezone PST -8
clock summer-time PDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00
clock save interval 12

On that note, if you’re having trouble with certificates, this should help, as I was just getting “% Error in saving certificate: status = FAIL” until I set the following:

debug crypto verbose
terminal monitor

There are more debug crypto commands too, some helped, some were just too verbose. This was probably the best cisco reference. This was useful in troubleshooting, as well as the event logs on the IAS server, which showed the wrong usernames and such. In one case my laptop, which isn’t in the domain, was sending ‘LAPTOPNAME\localuser’ without asking me, go to the wireless card, properties, ‘Wireless Networks’ tab, click on the AP profile, properties, Authentication tap, Enable 802.1x, Set EAP type to PEAP, properties. At the bottom under ‘Select Authentication Model’ choose ‘smart card or other certificate’ or ‘Secured Password’ then configure, both have a check box about asking you for a username or not.

I came into all of this half-configured, so theres probably more to it but hopefully I saved some people some troubleshooting.

As a side note, don’t delete these, heh:

aaa authentication login default local
aaa authorization exec default local

Removing the later while cleaning up was letting me login but was dropping me into level 1 access and giving me “% Error in authentication.” when I tried to enable. I was trying to clean up because there were piles of different authentication methods from previous attempts of multiple people to get this going.

update 12/18:

Figured out NTP, heh, ‘sntp server ipaddress’ in global config mode and ‘show sntp’ in exec mode.

Also ended up switching to WPA2 as Vista wasn’t liking the WEP and I really couldn’t tell why. Unfortunately this also means that XP needs the WPA2 patch. Ended up with this:

dot11 ssid YOURSSID
authentication open eap YOURGROUPNAME
authentication network-eap YOURGROUPNAME
authentication key-management wpa
guest-mode

interface Dot11Radio0
encryption mode ciphers aes-ccm tkip
ssid YOURSSID
! snip some default stuff

aaa group server radius YOURGROUPNAME
server RADIUSIP auth-port 1645 acct-port 1646

radius-server host RADIUSIP auth-port 1645 acct-port 1646 key 7 YOURSECRET

This is a good place to start.

I had to move a database to Microsoft SQL 2005 today. I managed to get both the servers to attach in SQL Management Studio (2005) and use the Copy Database Wizard (CDW) (Right click database, tasks, copy database) to set up a job (SQL Server Agent -> Jobs in the management studio) but it was failing. Event logs indicated it was because it was trying to log into the remote server as ‘DOMAIN\ANONYMOUS LOGIN’ or something stupid.

I tried with both the “detach and attach method’ where you have to provide a share path to the data files (I just used the administrative share on the source server) and the ‘sql management object method’. While I had domain admin credentials on both servers, they were in different domains (part of the reason of the migration). I think the failure was the attempt to use the “Integration Services Proxy Account” which probably is not setup at all, or obviously correctly.

Event log shows Source: SQLISPackage, Event ID: 12550:

Event Name: OnError
Message: Failed to connect to server SOURCESERVER.
StackTrace: at Microsoft.SqlServer.Management.Common.ConnectionManager.Connect()
at Microsoft.SqlServer.Dts.Tasks.TransferObjectsTask.TransferObjectsTask.OpenConnection(Server& server, ServerProperty serverProp)
InnerException–>Login failed for user ‘NT AUTHORITY\ANONYMOUS LOGON’.

I ended up detaching the database via the management stupid and copying the ldf and mdf files to the new servers data directory. Then running a query to attach it (one could probably do this visually). I got the query template from here rather than trying to remember how to type it. the “FOR ATTACH” part is important, heh.

‘Converting database ‘blah’ from version 539 to the current version 611.’ Showed up the query results, so it looks like it “upgraded” on it’s own.

This problem has bugged me for a while and I finally resolved it this week. I had a good idea of the cause but it just wasn’t important enough to solve. I wrote this up in a mailing list post, by I think the winpcap-users list is slow, or my antispam is causing trouble as it usually takes a great deal of time for me to get messages from the list.

Running Wiresharp 0.99.6a with Winpcap 4.0.1 on Windows Server 2003 x64 R2 SP2 wasn’t showing all the traffic. For the most part, I was only seeing the TCP handshake although occasionally I’d see another packet or two, such as the bindrequest in an LDAP query and nothing else. This is on Dell 1955 blades with Broadcom BCM5708S NetXtreme II GigE (NDIS VBD Client) chipsets. I assumed it was some kind of offloading like TCP checksum but more advanced. I eventually found that the Microsoft ‘Scalable Networking Pack’ was integrated into Server 2003 SP2, which includes ‘Chimney’ they’re advanced TCP Offloading Engine.

I had looked in the advanced driver properties and saw that ‘Large Send Offload’ and ‘TCP Check Offload’ were both disabled and gained a false sense of security from this. Apparently Chimney isn’t controlled through here.

‘Netsh int ip set chimney DISABLED’ Turns off chimney though. I don’t know that you want it on except on high performance webservers or what not, so it would have been nice if it wasn’t on by default. There’s a bunch of registry entries as well, but that command works without a reboot of any kind.

After an upgrade to Exchange 2007 SP1 from 2007 RTM, the Xerox WorkCentre Pro stopped sending email scans. I originally thought it was an LDAP problem, because I had seen an LDAP error the night before and WireShark on the DC was showing the bind request but no other traffic. I finally resolved the Chimney problem with winpcap, but in the interim just used a port mirror on the catalyst.

Attempts to Email Scan on the WorkCentre Pro were throwing out an error page that said ‘Job Status: FAILED Job must have at least one “To:” recipient.’ Under the ‘Message Settings:’ header it listed From: with a valid address but the To: line was empty. Getting wireshark going showed that it was a 5.7.1 SMTP error and the Xerox just wasn’t communicating as much. Whitelisting the IP for the Xerox on the SMTP server resolved the problem.

I picked up “Practical Packet Analysis: Using wireshark to solve real-world network problems” recently by Chris Sanders. Overall, it’s a little fluffy for my taste, I should know better than buying non ORA, AWP or Cisco Press books, but I’ve heard decent things about No Starch Press, somewhere.

If you’ve done any intermediate networking, the first chapter should be a quick review for you. If you’ve done any network troubleshooting with tcpdump/ethereal/wireshark, the next few chapters should be review as well. The rest of the book is mostly examples of traffic and how to tell what’s going on. This is nice, but at least once I felt the screenshots weren’t in the right order (DHCP DORA). All in all, I was really hoping this book would be more advanced than it was, but when you head in that direction, the author tells you to read an RFC.

There wasn’t any talk about TCP checksum offloading that I saw, or the heavier stuff like Chimney that microsoft is doing now that I don’t know anything about. Since I picked up the book because I’m having weird TCP segment timeouts and getting different traffic out of wireshark on the server and my workstation, I was hoping for more information that would be specific to this problem. Back to RFCs and google I guess.

If you consider yourself a senior linux systems administrator, most of this should be review for you. If you haven’t done much traffic analysis in the past, it’s worth picking it up and reading it. It’s a pretty light read, I didn’t skip any chapters but did skim a few and it was only a few hours of reading.

ISBN-13: 978-1-59327-149-7

I did finally get OMSA running fully on the Dell Poweredge 1955 blade with the SAS 5/iR (LSI SAS1068 PCI-X Fusion-MPT SAS) RAID controller (Not PERC5, apparently, which means its the mptsas driver and not megaraid_sas like PERC5). This confirmed my suspicion that ‘omreport storage controller’ listing the device as degraded was because the firmware was below the ‘minimum firmware’ version and not because of the virtual disk which ‘omreport storage vdisk’ listed as Ready and OK. Rainer Sabelka helped me out again noting that you could run the RHEL firmware update off support.dell.com.

The hard part was that I kept selecting RHEL5 in the pull downs, finding the utiltity and clicking “Download Now” which would send the standard Dell self-extracting executable which was very dos looking. I was thinking maybe you could take the firmware file and use some utility to upgrade it, whatever replaced omupgrade. Eventually I found that if you click on the file on dells site instead of download now, it gives you additional options, one of which is a BIN file for redhat. You can download this, chmod 755 it, and run it. It’ll less the release notes, then start the install when you say y. I shut down all the dell omsa stuff via init.d scripts before running this based on recommendations in the release notes. I think ‘file.bin –version’ brings up the notes such that ‘q’ just exits instead of continuing the install. Note that the install doesn’t force a reboot at the end, but recommends it. I didn’t see any changes in the omreport output until the reboot, so you’ll want to do the upgrade when you can reboot to avoid possible wonkyness.

As a side note, this is a decent starting place for Debian on Poweredge gear, although it says:

The three PowerEdge components that cause the most difficulty for new Debian sysadmins are the PERC 5 SAS RAID controllers (megaraid_sas driver), the SAS 5 non-RAID controllers (mptsas driver), and the Broadcom gigabit network adapters (bnx2 driver).

Which is a little confusing as the 1955 only has the mptsas driver running under etch, not the megaraid_sas, but it’s obviously doing raidish things and since we’re talking about linux and not windows I was thinking it wasn’t fakeraid. This is doubly interesting though:

vmware14:/proc/scsi# cat scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: FUJITSU Model: MAY2073RC Rev: D108
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: FUJITSU Model: MAY2073RC Rev: D108
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 01 Id: 00 Lun: 00
Vendor: Dell Model: VIRTUAL DISK Rev: 1028
Type: Direct-Access ANSI SCSI revision: 05

So it shows both disks, which are “raid1” and this “Virtual Disk”. Theres some jabber here about how it’s not RAID because it’s showing the disk and the other agrees and claims faulty assumptions, but there’s no reference or linkage. Here a Dell engineerish guy says “Yes the SAS 5/iR is just very light weight hardware RAID.”

Supposedly theres an ‘lsiutil’ package hiding in a driver package somewhere but I couldn’t find it in a brief search. The OMSA utilities appear to be working, although I still haven’t tried SNMP yet.

I’ve been playing with omsa on debian etch for a bit (dell 1955 blades) and wanted to post the notes on here. I’ve been using the sara.nl packages. After install, you’ve gotta start the daemon up: ‘/etc/init.d/dsm_om_connsvc start’.

Then you login as root/yourrootpassword. I’ve seen docs somewhere, I forget, that any user in /etc/passwd can login and those that are in the root group are admins. But it doesn’t work. If you check out /var/log/user.log you’ll see pam errors.

For some awesome reason when googling the dell list server archives you always get hits on the monthly gz digests but not the individual messages. I saw a few about this but I’m not going to dig them all back up, but this is good.

Basically you need 32bit versions of the following files and to put them in /lib32:

/lib/libsepol.so.1
/lib/libselinux.so.1
/lib/security/pam_unix.so
/lib/security/pam_nologin.so

You can figure out what package each file comes from with ‘dpkg -S file’, ie, ‘dpkg -S /lib/security/pam_unix.so’ which comes from libpam-modules. Its worth noting this package comes from the PAM master package so it it’ll be in pool/main/p/pam in the repo rather than pool/main/libp or whatever. Grab the i386 version of each package (libsepol1, libselinux1, libpam-modules) and extract them with dpkg -x package temp, then go into the temp tree (cd temp/lib) and copy the respective files to /lib32. I just copied the whole security folder from libpam-modules by the way. I used ‘dpkg -l | grep package’ to find out what version of each package was installed, then grabbed the same version in i386.

Run ‘ldconfig’ then tail /var/log/user.log just to keep an eye on things, and you should be able to log in now.

update: it’s worth linking that the sara.nl folks have revealed an wiki site with many of these answers here.

I had to move openldap from one box to another, different version.

The slap* tools that come with openldap appear to access the files directly while the ldap* tools in ldap-utils go through the ldapserver.

I started by exporting the old server to ldif with ‘slapcat -l oldserver.ldif’. Then copy this file to the new server

Here were some problems. I couldn’t slapadd because some of this already existed. I ran ‘dpkg-reconfigure slapd’ to get the config file to match what was on the old server. In retrospect I could have just copied the old config, but I wanted to make sure there weren’t new config options that were needed. Reconfigure asks you what OU and DNS entries you want to use and creates a new database. Of course, this will have admin entries, so you’ll still have this above problem.

Stopped slapd. I went into /var/lib/ldap and removed all the files. Then used ‘slapadd -l oldserver.ldif’ to import the entries, which created a new database. Make sure all these files are owned by the user that slapd will run as (chown -R openldap.openldap /var/lib/ldap).

Now start slapd and see if it’s running. the init.d script on debian etch was starting it, but it was closing with no errors. I changed the ‘loglevel’ line /etc/ldap/slapd.conf to ‘511’ and restarted, seeing “bdb_db_open: alock package is unstable” in /var/log/syslog before slapd shut down. Somehow I had gotten this created in all my screwing around. I removed alock and log.* from /var/lib/ldap, double checked the permissions which I think slapadd had also mangled at some point, and started the server. Came up fine now. Tested with ldapsearch -x -W -b ‘basedn’ where basedn is the basedn =P.

This helped me figure out some of the differences between slap* and ldap* and this made me realize that the “DB_KEYEXIST: Key/data pair already exists” error was because the administrative entry was already in the preconfigured database so I had to start from scratch.

I’ve got a dell blade (1955 or 1855 or something) that has a failed SAS disk and I need to get into the RAID array config without rebooting. The box runs debian etch 64bit. While dell has this crafty linux site with lots of info, it’s pretty rpm-based distro oriented (redhat/sles). There are some debian bits hiding in places though like omsa contributions here. (Be sure to go to IdeaStorm and vote for better debian support for OMSA here and here.) OMSA is the “Open Manage Server Administrator”. It looks pretty java oriented. I installed dellomsa_5.1.0-5_amd64.deb from here though, roughly following these directions.

It’s worth noting that the RAID is the new (PERC 5? PERC 6?) “generation” of SAS RAID, using the MPT Fusion drivers LSI. lspci provides “LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS (rev 01)” so I tried using the MegaCLI tools but there’s a lack of documentation and I had to download an rpm, extract it, and try trunning the MegaCli and MegaCli64 tools by hand, which both said I had no controllers (‘MegaCli64 -adpcount’ I think). This uses the ‘mptsas’ kernel module

Theres a qwoot dell wiki project that links to some docs on OMSA. The standard dell download interface totally pisses me off, as it’s table design is hard to read and often broken like here where a lot of the links just go to gpg signatures.

I couldn’t get logged into the web interface, which you start with /etc/init.d/dsm_om_connsvc and runs on port 1311. If you’re connecting and getting a “application/octetstream” MIME type download prompt, it’s probably because you’re not using ssl and it is, reconnect with ‘https://ip.ip.ip.ip:1311’. It’s pointed out here that password information is stored in /opt/dell/srvadmin/iws/config/iws.ini but doesn’t appear to be used. It looks like PAM is used, but I didn’t get so far as to debug PAM to make sure it was getting hit. Just try logging in with the root/rootpassword credentials.

I tried using omreport and whatnot in /opt/dell/srvadmin/oma/bin (use omhelp to get information on onreport, ie: omreport chassis info), but these tools said they couldn’t find any controllers or chassis info. I’m not sure whats msising there.

I removed those debs then installed dellomsa_5.1.0-2_i386.deb with ‘dpkg –force-architecture -i dellomsa_5.1.0-2_i386.deb’ as it’s i386 and this is amd64. Now ‘omreport chassis info’ brought back real data. ‘omreport system summary’ is useful, but ‘omreport storage controller’ still reports ‘No controllers found’.

Maybe I’ll go back to working with the megacli stuff. I wish I could get source instead of rpms.

Seriously people. What the hell is the use of a RAID controller that I’ve got to shut the box down to manage?

update: this looks common (1, 2, 3), I searched the linux-poweredge dell mailing list. Looks like an etch thing, mebbe an etch 64 thing. I just rebooted the box and rebuilt from the bios.

update2: Rainer pointed out on the poweredge list here that he needed to load mptctl before dataeng. so ‘ /etc/init.d/dataeng stop ; modprobe mptctl ; /etc/init.d/dataeng start’ got the storage portion of omreport working for me. Also Bas van der Vlies asked if I had tried their packages so I removed the i386 dellomsa, reinstalled the latest package from their distro on sara.nl (dellomsa_5.2.0-2_amd64.deb) and that worked alright. Interestingly ‘omreport storage controller’ says the array state is degraded. I put a new disk in last night and started the sync in BIOS, so I’m curious if it didn’t finish.

I still can’t login to the web portion, but this is good enough.

update3: noted that he’s arranged many of these notes in a secret clean wiki here, the FAQ has many of the same tips without my rambling. This includes the bits that I figured out from mail list posts on how to get the IWS (web) login working correctly, which I documented here.

This is on an ASA5520 running “Cisco Adaptive Security Appliance Software Version 7.0(6)”

aaa-server protocol nt , ldap, kerberos, radius or tacacs for active directory authentication? I thought googling for “aaa-server protocol nt” might bring up some decent results but instead I found one of my own posts in the first page of results. Fail.

As noted in the above linked post, NT is easy to set up, but I wonder if it’s supported by newer software. Apparently NT uses NTLMv1, bad.

It sounds like kerberos would work with active directory (I know AD has kerb likeness, but i never trust microsoft’s implementations of this crap to actually work in the same way as everything else, see the obfuscated SMB/CIFS protocol). There’s an interesting caveat with this implementation not supporting changing passwords during negotiation due to password expiration. I don’t think anything we use currently supports that anyways. I wonder if you can change expired domain passwords via OWA yet.

There are a number of examples (1, 2) out there using RADIUS, but that’s so old skewl. Cisco has some code / walkthrough (adsm) for doing kerb/ldap for authentication and authorization respectively here.

aaa-server mykerb protocol kerberos
aaa-server mykerb host 10.0.0.xx
kerberos-realm CORP.EXAMPLE.COM
aaa-server mykerb host 10.0.0.xy
kerberos-realm CORP.EXAMPLE.COM
aaa-server myldap protocol ldap
aaa-server myldap host 10.0.0.xx
ldap-base-dn ou=CORP_USERS
ldap-scope onelevel
ldap-naming-attribute uid
aaa-server myldap host 10.0.0.xy
ldap-base-dn ou=CORP_USERS
ldap-scope onelevel
ldap-naming-attribute uid

Then testing halfworks:

test aaa-server authentication mykerb host 10.0.0.xx username USER password PASS

This produces “ERROR: Authentication Rejected: Unspecified” when username and password are correct, otherwise “ERROR: Authentication Rejected: Invalid password” or “ERROR: Authentication Rejected: User was not found”. So something’s working right, but I thought a computer account needed to exist for kerb.

The one Cisco page notes two things to check in troubleshooting, 1) clock skew (normal for kerb, I setup ntp already) and 2) “pre-authentication”. They fail to explain the latter. I found the information I needed here. “0x19 – KDC_ERR_PREAUTH_REQUIRED: Additional pre-authentication required”. I take it that windows sends this kerb error if you’re trying to do kerb but aren’t authenticated, say via a computer account. Meh. Checking the “Do not require kerberos preauthentication” box on the account tab of the active directory account brought back a success in the above cisco test command, but I’m not sure that the tradeoff is acceptable. Other cisco access server software has you configure an account for it, however complex the process looks.

Since this is only DES from all I can tell, which makes everyone gawk, but I can’t find any evidence that Windows Kerberos support 3des, whilst I can find evidence of people having to switch from 3des to des to make other kerberos implementations work with Windows. ktpass, a windows utility for fscking around with kerberos has these options:

[- /] crypto : DES-CBC-CRC : for compatibility
[- /] crypto : DES-CBC-MD5 : for compatibliity
[- /] crypto : RC4-HMAC-NT : default 128-bit encryption

So I’m not feeling like kerberos is really a huge step up as far as password security goes. It’s certainly progress over NTLMv1 and it’s low-entropy retardedness, but RADIUS just might be the right choice here. WinKerb would use RC4-HMAC, but I’m sure it’s using DES to talk to the Crisco.

Ken Caruso pointed out some commands in IOS for configuring preauth, but they don’t appear to exist in ASA v7.0.

So, I guess it’s trusting RADIUS + MD5.

It does appear that ASA v8.0 supports NTLMv2, perhaps other features? Worth upgrading? Hmm.

this took far too long to find, longer than a walk to the data center. I’ve removed the serial numbers to protect the innocent. I just wanted to list card models without reading them off the card.

sw01#show inventory
NAME: “WS-C6509”, DESCR: “Cisco Systems Catalyst 6500 9-slot Chassis System”
PID: WS-C6509 , VID: , SN: xxxxxxxxxxx
NAME: “5”, DESCR: “WS-X6148-GE-TX 48 port 10/100/1000mb EtherModule Rev. 6.1”

I removed a bunch of lines there. I was trying to figure out why “switchport trunk encapsulation dot1q” wasn’t an option (there was no encapsulation option) on this card.

Finally found on Page 11 of this that these cards don’t support ISL, so they default to 802.1q:

WS-X6502-10GE
WS-X6548-GE-TX, WS-X6548V-GE-TX, WS-X6548-GE-45AF
WS-X6148-GE-TX, WS-X6148V-GE-TX, WS-X6148-GE-45AF

Which was important because I was seeing traffic on the host that didn’t appear tagged, and 802.1q doesn’t tag the native vlan (vlan 1) by default. Of course I have no idea what ISL traffic would look like in a tcpdump, probably thrown away as garbage, but who knows, better to see and know than to guess.

I like seeing things happening. If you’ve never tested SMTP by sending a test message by hand with HELO/MAIL/RCPT/DATA, you really should give it a shot. You can use mailsend or such to do this testing somewhat by hand, but when things fail I find it easier to see it directly.

‘EHLO domain’ via smtp should list what AUTH options are available when testing SMTP AUTH (which lets you use authentication to allow relaying). Sometimes this is different when you’re over SSL (‘250-AUTH GSSAPI NTLM LOGIN’) than when you’re not (‘250-AUTH GSSAPI NTLM’), for good reason such as some of the AUTH options aren’t encrypted or are simply obfuscated (read: base64).

openssl s_client has built in functionality for testing smtp:

openssl s_client -starttls smtp -crlf -connect 1.2.3.4:25

But when connecting to an exchange server, it just hangs at CONNECTED. Trying again with the ‘-debug’ flag shows why:

write to 080B01E8 [BFFFED30] (10 bytes => 10 (0xA))
0000 – 53 54 41 52 54 54 4c 53-0d 0a STARTTLS..
read from 080B01E8 [080AA400] (8192 bytes => 28 (0x1C))
0000 – 35 30 33 20 35 2e 35 2e-32 20 53 65 6e 64 20 68 503 5.5.2 Send h
0010 – 65 6c 6c 6f 20 66 69 72-73 74 0d 0a ello first..

There’s chatter and bugs (1,2) about this out there.

My openssl versions are ancient (such as OpenSSL 0.9.7a Feb 19 2003 (‘openssl version’)), so nobody else will likely have this problem anymore because they added code in ‘openssl-0.9.8e/apps/s_client.c’ to send a ‘EHLO openssl.client.net’ before starttls. On second thought, ubuntu feisty is still on 0.9.8c, so mebbe you will.

Once connected, you can try authing. If you’re using AUTH PLAIN, which I have no idea if exchange supports this but others due, you’ll want to encode your username and password with base64 something like this (thanks crash), I think:

printf “\0username\0password” | openssl enc -a

Then send this in the smtp session with ‘AUTH PLAIN base64’ where base64 is whatever you get out of openssl. It’s worth noting that you’ll get different results if you use “echo -n” instead of printf. crash says to use ‘echo -ne’ to get escape characters working right. YMMV.

Testing ‘AUTH LOGIN’ is similar. Send ‘AUTH LOGIN’ and you’ll get back ‘334 VXNlcm5hbWU6’ which says ‘334 Username:’ when you decode the crap with base64 with ‘printf VXNlcm5hbWU6 | openssl enc -a -d’. Encode your username with ‘printf “username” | openssl enc -a’ and paste this back to the server. You’ll get a ‘334 UGFzc3dvcmQ6’ response which again is ‘334 Password:’. Send your password back in the same manner, base64 encoded (not encrypted, heh. that’s what the ssl is for).

If all works you’ll get something such as ‘235 2.7.0 Authentication successful’ in response. Remember that sometimes throwing -debug on the end of openssl gives you more information. Now try your MAIL/RCPT/DATA shit and make sure you can still relay.

I’d love to provide a way to test NTLM, but it appears, no surprise, to not be that popular. Exchange 2007 will support LOGIN. Open EMC. Server Configuration -> Hub Transport (Or Edge Transport on an edge server). Under Receive Connectors right click and go to properties for the connector. On the Authentication tab make sure “Basic Authentication” is checked and make sure to check the following box limiting to only allow this after starttls.