While several blog posts have shown how to retrieve credentials through this vulnerability, we decided to dig deeper and see whether it was possible to execute arbitrary code through this issue.
DISCLAIMER: This blog post was written a year and a half ago and we have postponed publication upon Veeam’s request, but given a recent post from Watchtowr (here) detailing the almost exact vulnerability, we feel like we can now freely publish this article.
The original statement released by Veeam regarding this vulnerability indicates that it allows an attacker to gain access to encrypted credentials from the server (https://www.veeam.com/kb4424). Quickly thereafter, researchers showed that it was actually possible to retrieve unencrypted credentials as well. Given the widespread use of Veeam and the fact that backing up systems has taken precedence over updating them, we thought it would be practical for our pentesting team to have a way of exploiting the vulnerability.
So while many hackers were busy trying to exploit our challenges at Insomni’hack, we were busy attempting to produce a PoC for this vulnerability. We started analyzing version 11.0.1.1261_20220302 and while my colleague itm4n was able to quickly reproduce the vulnerability, several other blog posts had already been written explaining the vulnerability and how it can be exploited, so we’ll refrain from repeating that information here. Instead I invite you to check out the following articles:
- https://www.horizon3.ai/veeam-backup-and-replication-cve-2023-27532-deep-dive/
- https://www.huntress.com/blog/veeam-backup-replication-cve-2023-27532-response
In a nutshell, Veeam’s Backup Service allows unauthenticated requests to a WCF endpoint which allows amongst other things to retrieve any credentials stored by Veeam. All the previously written articles we could find stopped at recovering credentials while only quickly touching on the fact that there are many (thousands) of other endpoints which can be called and many of them happen to deserialize C# objects. Deserialization of user-supplied input has proven to be tricky and can often lead to remote code execution, so we thought it would be worth while searching for whether this might be achievable in this specific scenario. Initially, we thought we may be able to execute code through any of the following ways (in the order of easiest to exploit to hardest for someone who has never used Veeam before):
- Arbitrary .Net deserialization
- Injection in stored SQL procedures to hopefully call
xp_cmdshell - Exploitation of legitimate functions which happen to execute code
We started off by looking at the .Net serialization and it turns out that Veeam uses the BinaryFormatter to serialize and deserialize data (at least within the context of this vulnerability). Past research indicates that this is particularly dangerous, as even when using a custom SerializationBinder, there can still be ways of executing code.
So we quickly spun up ysoserial.net and attempted to force the server to deserialize an object which would result in code execution. This failed with the following output in the server logs:
[04.04.2023 01:29:00] <27> Error Deserialization of 'System.Security.Claims.ClaimsPrincipal, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' is not allowed. Unable to deserialize System.Security.Claims.ClaimsPrincipal, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
[04.04.2023 01:29:00] <27> Info 3322 types allowed. Similar allowed types:
[04.04.2023 01:29:00] <27> Error Binary deserialization failed
[04.04.2023 01:29:00] <27> Error Deserialization of System.Security.Claims.ClaimsPrincipal, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 is not allowed (System.NotSupportedException)
[04.04.2023 01:29:00] <27> Error at Veeam.Backup.Common.CWhitelist.EnsureIsAllowed(String afqn)
[04.04.2023 01:29:00] <27> Error at Veeam.Backup.Common.RestrictedSerializationBinder.ResolveType(ValueTuple`2 key)
This didn’t look good, and digging through the code, we quickly confirmed that a whitelist of authorised classes is used when deserializing data.
protected override Type ResolveType(
[TupleElementNames(new string[] {"assemblyName","typeName"})] ValueTuple<string, string> key
) {
this.EnsureTypeIsAllowed(key);
Type type = base.ResolveType(key);
RestrictedSerializationBinder.CheckIsRestrictedType(type);
return type;
}
// Token: 0x06001236 RID: 4662 RVA: 0x000324B8 File Offset: 0x000306B8
private void EnsureTypeIsAllowed(
[TupleElementNames(new string[] {"assemblyName","typeName"})] ValueTuple<string, string> key
) {
if (!this._serializingResponse && SOptions.Instance.ShouldWhitelistingRemoting)
{
this.EnsuredBlackWhitelistsAreLoaded();
string afqn = key.Item2 + ", " + key.Item1;
if (this._mode == RestrictedSerializationBinder.Modes.FilterByWhitelist)
{
RestrictedSerializationBinder._allowedTypeFullnames.EnsureIsAllowed(afqn);
return;
}
if (this._mode == RestrictedSerializationBinder.Modes.FilterByBlacklist)
{
RestrictedSerializationBinder._notAllowedTypeFullnames.EnsureIsAllowed(afqn);
}
}
}As we can see, there is also the possibility of using a blacklist instead of the whitelist, but by default the whitelist mode is used.
public RestrictedSerializationBinder(
bool serializingResponse,
RestrictedSerializationBinder.Modes mode = RestrictedSerializationBinder.Modes.FilterByWhitelist
) {
this._serializingResponse = serializingResponse;
this._mode = mode;
}Having never searched for deserialization gadgets in .Net before, we thought it would be interesting to give it a try, so we spent a large chunk of Insomni’hack’s CTF combing through the various classes in the whitelist and searching for interesting functions, but this did not result in anything useful. I’ll blame sleep deprivation and lack of tooling as the main culprits.
We then shortly entertained the notion of looking through the thousands of accessible endpoints in search of some which may allow for code execution. I’ll admit I only scanned through the function names in search of anything referencing code evaluation or execution but didn’t find anything relevant. We did however notice that many functions just end up calling a stored SQL procedure, so we continued looking through the stored procedures in search of injection possibilities but once again nothing evident came up.
We then decided to turn back to the deserialization issue and instead of searching for a whitelisted class that executed interesting code, we checked whether the blacklist actually prevented the use of all the ysoserial gadgets. So we created a small .Net project re-implementing the custom deserialization routine and quickly discovered that the ToolboxItemContainer can be used to execute arbitrary code despite the blacklist.
Having discovered this, we went back to the Veeam code in search of places where the blacklist might be used instead of the whitelist. The only time this seems to happen is if the following function is called directly within the CProxyBinaryFormatter class.
public static T Deserialize<T>(string input)
{
T result;
try
{
byte[] serializedType = Convert.FromBase64String(input);
BinaryFormatter deserializer = new BinaryFormatter
{
Binder = new RestrictedSerializationBinder(false, RestrictedSerializationBinder.Modes.FilterByBlacklist)
};
result = CProxyBinaryFormatter.BinaryDeserializeObject<T>(serializedType, deserializer);
}
catch (Exception ex)
{
Log.Exception(ex, "Binary deserialization failed", Array.Empty<object>());
throw;
}
return result;
}This method is actually called quite a bit throughout the code base, but only seemed to be called once in the exposed WCF endpoints.
Deserialize functionAt this point we got pretty excited as it seemed like a relatively straightforward affair to just call the vulnerable method with our serialized object. Unfortunately, we hit a roadblock which slowed our exploitation attempts. When calling the vulnerable function, before serializing any of our input, the application will in this case actually check that the SessionContextId is valid.
private CRemoteInvokeRetVal ExecuteStartAgentSessionTrafficProxy(CStartAgentSessionTrafficProxyRemoteInvokeSpec spec)
{
this.GetExistingSessionContext(spec.SessionContextId);
return new CStringInvokeRetVal(
this._managers.AgentDispatcher.GetProxyServerConnection(
spec.AgentId, spec.SerializedConnectionParams,
spec.ProxyHostId
)
);
}Not knowing how this session identifier is generated or even what it related to meant that some more reverse engineering was required. So following the different calls down the GetExistingSessionContext function, we eventually end up at the following piece of code in the CRestoreSessionContextScope class.
public CRestoreSessionContext FindBySessionId(Guid sessionId)
{
object @lock = this._lock;
CRestoreSessionContext crestoreSessionContext;
lock (@lock)
{
foreach (KeyValuePair<Guid, CRestoreSessionContext> pair in this._contextIdToContext)
{
Guid guid;
pair.Deconstruct(out guid, out crestoreSessionContext);
CRestoreSessionContext crestoreSessionContext2 = crestoreSessionContext;
if (crestoreSessionContext2.RestoreSessionId == sessionId)
{
return crestoreSessionContext2;
}
}
crestoreSessionContext = null;
}
return crestoreSessionContext;
}Apparently, the existing context identifiers are stored in memory in a variable named _contextIdToContext. The obvious next step was to figure out how we could write to this KeyValuePair. Looking a little further up in the code, we find the Open function which does exactly this:
public Guid Open(Guid restoreSessionId)
{
CRestoreSessionContext crestoreSessionContext = new CRestoreSessionContext(restoreSessionId);
object @lock = this._lock;
lock (@lock)
{
this._contextIdToContext[crestoreSessionContext.Id] = crestoreSessionContext;
}
return crestoreSessionContext.Id;
}Using dnSpy’s Analyzer, we can work backwards to figure out whether we can actually call this function from the exposed unauthenticated WCF interface. Thankfully this was easier than expected as there aren’t that many calls to the function, and we end up discovering that there is indeed a WCF endpoint named OpenVbRestoreSession which calls a function aptly named ExecuteOpenClientSession, as shown below.
The function itself is reproduced below.
private COpenVbClientSessionInvokeRetVal ExecuteOpenClientSession(COpenVbClientSessionInvokeSpec spec)
{
CRestoreSession crestoreSession = CRestoreSession.Get(spec.RestoreSessionId);
if (spec.MountRestoreSessionIdOrEmpty == Guid.Empty)
{
this._managers.ItemRestoreManager.OpenSharedSessionContextIfNotExists(
crestoreSession.LeaseId
);
}
else
{
CRestoreSession crestoreSession2 = CRestoreSession.Get(
spec.MountRestoreSessionIdOrEmpty
);
this._managers.ItemRestoreManager.AttachSharedSessionContext(
crestoreSession.LeaseId,
crestoreSession2.LeaseId
);
}
return new COpenVbClientSessionInvokeRetVal(
this._managers.ItemRestoreManager.OpenSessionContext(
spec.RestoreSessionId
),
SProduct.Instance.ProductVersion, TimeZoneInfo.Local
);
}The new context is created at the bottom of the function and unfortunately there are some additional hurdles that need to be overcome before getting to that point. In particular, the first line of the function verifies that the RestoreContextId we specify in the request actually exists. Hoping we wouldn’t have to do this too many more times, we once again searched for where these identifiers are stored and how to generate one. In this case, they happen to be found in the SQL database and I’ll spare you the details of how we got to this (in part because I didn’t write it down and can’t remember it all) but they can be generated by calling a WCF endpoint named RestoreJobSessionsDbScopeCreateSession:
private CRemoteInvokeRetVal ExecuteRestoreJobSessionsDbScopeCreateSession(CCommonInvokeSpec spec)
{
CRestoreSessionInfo session = this._deserializer.DeserializeCustom<CRestoreSessionInfo>(
spec.GetParamAsString("session")
);
CDBManager.Instance.RestoreJobsSessions.CreateSession(session);
return CCommonInvokeRetVal.Create();
}This function is pretty straightforward, and it will simply create a new CRestoreSessionInfo object and add it to the database. All that needs to be done now is to serialize a valid object of that class and send it to the application to get our coveted RestoreContextId. The code below will achieve just that.
Guid jobid = Guid.NewGuid();
Guid multirestoreid = Guid.NewGuid();
Guid oibld = Guid.NewGuid();
Guid parentSessionId = Guid.NewGuid();
AccountSid asid = new AccountSid();
CRestoreSessionInfo abc = CRestoreSessionInfo.CreateNew(
EDbJobType.AmazonRestore, "jobname", jobid, "options", asid, "initName", "reason",
CPlatform.AzureCompute, CPlatform.AzureCompute, multirestoreid,
CRestoreSessionInfo.ERestoreType.SingleRestore, oibld, true,
"vmDisplayName", DateTime.Now, 1, parentSessionId
);
abc.LeaseId = Guid.NewGuid();
Console.WriteLine("Restore Session ID : " + abc.Id);
MemoryStream ms = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
string outputValue = CProxyBinaryFormatter.Serialize(abc);The parameters are quite arbitrary as long as they are of the right type.
We now have all the steps required to finally be able to hit our deserialization endpoint. In order we must therefore call the following WCF functions:
- RestoreJobSessionsDbScopeCreateSession
- OpenVbRestoreSession
- ExecuteStartAgentSessionTrafficProxy
It is therefore possible to execute arbitrary code on the server without requiring any privileges, which slightly changes the CVSS score of 7.5 which was initially attributed to this vulnerability.
After discovering the issue, we went through the patch to see how the initial vulnerability had been corrected and to determine whether the deserialization issue was still present. Having installed a bright new and shiny version 12.0.XXXXX of Veeam Backup and Replication, we spun up dnSpy on the new release and searched for how the application had been modified. This quickly led to the following code:
public string Invoke(string scope, string method, string parameters)
{
string result;
try
{
Log.Debug(string.Concat(new string[]
{
"Invoke: scope '",
scope,
"', method '",
method,
"'"
}), Array.Empty<object>());
Thread.CurrentPrincipal = new WindowsPrincipal(WindowsIdentity.GetCurrent());
XmlNode specNode = CRemoteInvokeSpec.GetSpecNode(parameters);
CAuthToken authToken = CRemoteInvokeSpec.GetAuthToken(specNode);
this._tokenValidator.Validate(authToken);
result = this.ProcessCommand(scope, method, specNode).Serialize();
}
catch (Exception exception)
{
CBackupSecureServiceErrorHandler.LogAndThrowFaultException(exception, scope, method);
throw;
}
return result;
}Each request to a WCF endpoint must now contain an authentication token, which takes the form of a JWT token which is validated in the following way:
X509SecurityKey issuerSigningKey = new X509SecurityKey(this._certificate);
TokenValidationParameters validationParameters = new TokenValidationParameters
{
ValidateAudience = false,
ValidIssuer = "Veeam",
ValidateIssuer = true,
ClockSkew = TimeSpan.Zero,
ValidateLifetime = false,
IssuerSigningKey = issuerSigningKey,
ValidateIssuerSigningKey = true
};
SecurityToken securityToken;
ClaimsPrincipal claimsPrincipal = new JwtSecurityTokenHandler().ValidateToken(authToken.RawData, validationParameters, out securityToken);I’m not sure why they decided not to validate the audience or the lifetime or the token, but it is signed with the server’s certificate and the signature is verified appropriately. So unless there is a way to force the server to generate a valid JWT token for another application or to compromise an old one in logs somewhere, the solution seems acceptable.
We then turned our attention to the deserialization issue and quickly verified that nothing had changed in that respect. So an authenticated user can still execute arbitrary code on the server. Not knowing the inner workings of Veeam, it is difficult to say how impactful this is, as it is very possible that the privileges required to exploit the vulnerability allow to legitimately execute code on the server. Nevertheless, we reported the issue to Veeam who deemed that it warranted an update (version 12.0.0.1420_20230413).
Once again we dug through the code to see what had changed in the newest release. The only notable difference was the addition of the ToolboxItemContainer to the blacklist. This seemed like somewhat of a lazy reaction, as it did indeed prevent our PoC from working, but the use of the whitelist would have been preferred. Unsurprisingly, with a little more digging, we found that the ObjRef gadget could still be used to execute code on the server.
This gadget is similar to the UnicastRef Java gadget, which essentially transforms the target into a remoting proxy which will connect to an attacker-controlled URL with .NET remoting. It is then possible to entirely bypass the blacklist and use any other gadget to compromise the server. In our case, we used the RogueRemotingServer from CodeWhiteSec for this purpose.
We reported the issue with the patch and have been waiting ever since for an update from Veeam who requested that we do not publish a blog post in the mean time.
Fast-forward 17 months, and it seems like Veeam have finally decided to add the ObjRef gadget to the blacklist as well (see details about reverse engineering the latest patch here from watchtowr). We’re finally publishing this blog post since most of the vulnerability details are already provided in watchtowr’s blog. Here is also a full disclosure timeline for those who are interested:
- 03.04.2023 – Initial disclosure to Veeam
- 03.04.2023 – Veeam accepts the vulnerability submission
- 14.04.2023 – Patch 20230413 is released
- 17.04.2023 – We indicate to Veeam that the patch can be bypassed with the
ObjRefgadget - 04.05.2023 – Request an update on the issue from Veeam
- 09.05.2023 – Veeam indicate substantial code change is required and a different formatter will be used for next major release
- 27.06.2023 – Request information on when next release might be available
- 29.06.2023 – Response from Veeam that it is under development and request to not publish a blog post yet
- 29.01.2024 – Request an update on the issue from Veeam
- 29.04.2024 – Response that work is still in progress
- 16.08.2024 – Request an update from Veeam indicating we will publish a bog post in September
- 19.08.2024 – Veeam ask if we can test the latest release and request an email to share it with us
- 03.09.2024 – We provide email but are still waiting for the patch files
- 09.09.2024 – Watchtowr publish blog post analyzing another more recent patch which clearly covers parts of the vulnerabilities we had discovered