5

I have a problem :)

I would like to identify how much change there has been at block level to either an entire disk or a particular file.

Ideally there is a tool that runs a scan on a disk or a file and produces "fingerprints" for each block and then is able to compare those fingerprints at a later stage to arrive a number of changed blocks since the last scan.

Does anyone have any ideas? Can anyone suggest any tools, command line utilities?

UPDATE

I found the following project:

https://github.com/OctopusDeploy/Octodiff

which is a C# implementation of the rdiff tool which is itself based on the rsync algorithm and essentially is along the lines of what I was hoping for.

Now what I need is a tool that outputs the data changed stats without writing out a delta file.

Bruno
  • 213
  • 2
  • 12
  • What's the use case? Or to put it another way, what's the problem you;re trying to solve? – BlueCompute Nov 12 '15 at 10:41
  • We are trying to figure out what amount of new data is generated at by an office that uses a local file server. We are contemplating moving this to a central location and want to understand the impact on the network. Some of that can be ascertained from how much data is being written to disk. A list of modified files is not suitable. – Bruno Nov 12 '15 at 12:49
  • `A list of modified files is not suitable` Why not? – BlueCompute Nov 12 '15 at 13:02
  • Because what we are replacing it with is a tool that pushes modified blocks upstream rather than entire files to minimize network traffic. Therefore to better predict what traffic it will generate looking at changed blocks will be more accurate. – Bruno Nov 12 '15 at 13:52
  • If you are trying to gauge total "network impact" why are you focused only on "changes"? Network impact will be the product of several factors: Example: (number of clients) X (file size) X (number of access attempts). You might want to consider consider logging performance counters on the server over the course of the day. If your concern is CIFS file sharing: `SMB Server Shares`, `Data Bytes/sec`, `_Total` – Clayton Oct 12 '16 at 15:31
  • If the actual server is network attached, i.e. SMB, you can check the network utilization there – Fredi Oct 12 '16 at 15:35
  • @Craig620 That won't work. I am interested in measuring the block changes that have happened on the disk. If I read a file from an SMB share 10 times I still have no change on the file apart from last access date. Measuring Data Bytes/sec will give me 10 x file size but not the amount of block change. – Bruno Oct 12 '16 at 22:31

6 Answers6

3

Microsoft block tracking feature similar to VMware CBT mechanism should be available in Windows Server 2016. You could access this feature by using some public API's (like Veeam employs vSphere CBT). So I guess it may help you to track blocks of the data that have been changed. We are also running hyper-v using veeam and sitting on Starwind vSAN that works well with VSS. I know Nimble supports VSS/CBT too.

batistuta09
  • 8,981
  • 10
  • 23
3

@batistuta09 mentioned Starwind https://www.starwindsoftware.com/starwind-virtual-san-free which is actually a virtual SAN and is free. It installs natively on Hyper-V and it has a journaling filesystem device called LSFS https://www.starwindsoftware.com/vm-centric-storage-lsfs that is VM oriented.

The nice thing about it that it generates new files (or chunks) sized AFAIK from 128MB to 512MB continuously when new data is written onto it. That might be sufficient for you to easily see how much "new" data is generated throughout a given period of time. The result might be not very precise though but probably will be good enough for your specific task and the installation and configuration is really simple.

Community
  • 1
Net Runner
  • 6,169
  • 12
  • 34
2

I'm not going to get into too many specifics, and I realize this is a Windows Server question, but I'll point you in two directions I think may be viable for your purposes.

1) You could use a journaling file system, and capture the journal entries as they happen to produce a not perfect, but very close approximation of how many changes have been made in a particular time slice. Then you could use some math to determine block level changes to a decent level of accuracy.

Many journaling file systems allow you to place the journal itself on an entirely different piece of storage than the data, so you could conceivably do that (with XFS for example) onto a large piece of storage, and then analyse the journal over time slices with a separate tool or watcher.

Lots of possibilities with this approach I am not mentioning.. the sky's the limit.

2) If you require an actually accurate map on time slice (1) vs time slice (2) at the block level in terms of changed/not changed, I might think about using DRBD and two separate read only slaves (or software RAID 1, which would be easier, and possible on Windows Server, but more physically limiting).. one which is brought in at Time (a) and then disconnected. Then bring the second one in at time (b) and then disconnect it, and then do a static compare using any sort of tool you wish.. you just need to read in each block from both sources and do an XOR. If the result is greater then zero, it changed.

T. B.
  • 323
  • 1
  • 2
  • 8
  • The idea is that at the end of the exercise I will have an answer like "x blocks have changed". I am guessing that the journaling can't give me the no of bytes changes in the file. – Bruno Oct 12 '16 at 22:34
  • The journal may not have the exact count of bytes changed, but it should have the exact number of blocks changed. – chicks Jan 03 '17 at 19:46
0

My suggestion would be to use a VM technology such as CBT in vSphere, or KVM using a backing file and overlay for storage.

Then you can leverage CBT or examine the overlay qcow2 image to determine what blocks changed.

Alternatively, if you're doing this to determine rate of change for replication traffic, use performance counters to examine total amounts of data written to the device.

MikeyB
  • 39,291
  • 10
  • 105
  • 189
  • 1
    CBT looks really good but unfortunately the Windows VMs I am working with are hosted on Hyper-V. Regarding performance counters they won't help me much. For example, I want to measure change between 9am and 9pm but if the file has been saved 100 times I won't get the number of bytes changed but the number of bytes changed * 100. – Bruno Oct 15 '16 at 15:36
  • Hyper-V also has sort of a CBT in WS2016. http://windowsitpro.com/blog/microsoft-brings-resilient-change-tracking-rct-windows-server-2016 – BaronSamedi1958 Oct 28 '16 at 12:12
0

One option could be to use ZFS and its snapshot feature. Make a snapshot on time X, and check what the snapshot space usage is at time Y. This will tell the size of differences between times X and Y.

ZFS snapshot space usage is described at Oracle ZFS documentation.

Tero Kilkanen
  • 36,796
  • 3
  • 41
  • 63
0

You can enable Shadow copies on all the volumes you want to monitor, then you can check the Used Shadow copy storage space with:

vssadmin list shadowstorage

The difference in the values between two runs of the command should give you a good estimate of the size of accumulated changes

Silvio Massina
  • 476
  • 2
  • 5