Toad World® Forums

ideas on configuring Subversion repositories?


#1

Hi, I’m evaluating using Subversion with Toad and I’m trying to figure
out a reasonable fashion to configure the repository, keeping it as
simple as possible, and I’m looking for suggestions others might have?
We have a production database and 4 development/test databases that are
cloned from production for various purposes. We are a private college
and use the SCT Banner HE package. We maintain a few modifications to
the Banner delivered code, and also a number of custom written add
ons–so we have package, view, table, etc, source that we wish to maintain.

o Do we set up one repository for our source, a production and test
repository, or a repository for each database instance?

o What kind of directory structure works best for PL/SQL
development?..seperate directories for different schemas?..different
object types?

o What happens when we clone our production database to our
develop/test instances if we use Toads Team Coding?

o Will Team Coding work for our SQL Nav users?..when I try to check
out a file in SQL Nav it tells me the version of Team Coding is too old
even though I just installed it using the servers side objects wizard in
Toad 10.6.


#2

Hi Wendy,

Hi, I’m evaluating using Subversion with Toad and I’m trying
to figure out a reasonable fashion to configure the repository,
keeping it as

simple as possible, and I’m looking for suggestions others might
have?

Keep it simple! Always the best advice.

If you read Subversion manuals, or look at many repositories, there is a
“standard” setup whereby you can either:

  • have one single repository with all your projects in; or
  • have one repository per project.

Beneath the above, you are advised to have three folders:

  • trunk
  • tags
  • branches

All current version support is development is done in trunk, development
on the next version is done in a separate branch in branches, and when a
release is made to customers, a “copy” is created in tags.

I use this setup myself, as follows:

c:\oracle\scripts\TOAD\trunk
c:\oracle\scripts\TOAD\tags
c:\oracle\scripts\TOAD\branches

As you can see I have a separate repository for each project.

We have a production database and 4 development/test
databases that are cloned from production for various purposes.

So far, one repository is all you need. Each different
database/development team can check out the latest code from the
repository. The development teams might require a branch to do
development work though - to avoid messing up code already in
production, but then again, code already in production should have a
tag. Equally, it should be in trunk as well.

So your repository admin would create a copy of the current trunk and
tag it as “Release 6.66” (or whatever) and put it in the tags directory.
You now have a base copy of the 6.66 release that is going to
production. This can be used for bug fixes in this release but without
necessarily affecting development work on-going in trunk on the next
release.

We are a
private college and use the SCT Banner HE package. We maintain a few

modifications to the Banner delivered code, and also a number of
custom written add

ons–so we have package, view, table, etc, source that we
wish to maintain.

o Do we set up one repository for our source, a production and test
repository, or a repository for each database instance?

One repository, I think. Your production code would be checked in
(imported) and immediately, a tag created in the tags directory as your
current base release of the production code.

Your production support team would work to fix bugs in this code,
probably in a copy of the base release in branches. All work they do to
fix bugs etc would be committed back to the branches part of the
repository.

Any development work would take place either in trunk (ie, starting from
the base release and going forward) or in a separate branch - assuming a
whole new start, or similar, exercise.

o What kind of directory structure works best for PL/SQL
development?..separate directories for different
schemas?..different object types?

I do all mine in one directory structure, I rarely have separate schemes
but if I do, I’d set up as follows (assumes trunk = development area):

c:\oracle\scripts\TOAD\trunk\schema_a
c:\oracle\scripts\TOAD\trunk\schema_b

All my work would take place there. When checking out my working copy,
I’d simply check out trunk - unless I’m only working on one particular
schema - do my work, make commits etc.

When it’s ready for testing teams to take over, I create a tagged
“pre-release” in the tags directory which the test team export/check out
and play with. Any faults or bugs are fixed in trunk as normal and then
applied to the tagged pre-release.

Eventually, the release is passed as suitable for production and I
create a new Release tag and from that, create a release kit (if
necessary) for the production team (which is me!) to apply.

o What happens when we clone our production database to our
develop/test instances if we use Toads Team Coding?

I don’t use Toad’s Team Coding. So I don’t have this problem. Under Toad
10.5 I use the Tortoise utility as my Third Party File Based Source
Control Provider. I also tend to use all the Tortoise commands outside
of Toad. Simply because I edit separate files in Toad to develop or bug
fix and I like to make sure that they all get committed as part of the
same (Subversion) transaction - rather than checking in one file at a
time from each tab.

I notice at 10.6, that I can also use the Subversion command line
utilities as a team coding provider - however, it appears that the
default settings for the subversion options are to “use locking on check
in/checkout” - this is against how Subversion works and is best avoided.

There is no need to lock a file on checkout and unlock it on check in -
so make sure the option is off.

o Will Team Coding work for our SQL Nav users?..when I try to check
out a file in SQL Nav it tells me the version of Team Coding
is too old even though I just installed it using the servers side
objects wizard in Toad 10.6.

Sorry, I don’t use SQL Nav, however, if it too uses Subversion as the
source control utility, then it should be able to work happily alongside
Toad’s own use of the SVN command line and, if you have it, other
people’s use of TortoiseSVN and other SVN tools.

I rather suspect you need to bypass the Toad Team Coding stuff on the
database. With Subversion, I really don’t think it’s needed, and I’ve
used Subversion with Toad for a long time, and I’ve never created any
database objects with the Toad server side wizard - I’m not allowed to
for a start! Only application tables and objects are allowed on our
databases.

HTH

Cheers,
Norm. [TeamT]

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk


#3

Hi Norm, thank you for such a clear response–this is very helpful. Wendy

On 10/22/2010 5:27 AM, Dunbar, Norman (Capgemini) wrote:

Hi Wendy,

>> Hi, I'm evaluating using Subversion with Toad and I'm trying
>> to figure out a reasonable fashion to configure the repository,
keeping it as
>> simple as possible, and I'm looking for suggestions others might
have?

Keep it simple! Always the best advice.

If you read Subversion manuals, or look at many repositories, there is a
"standard" setup whereby you can either:

* have one single repository with all your projects in; or
* have one repository per project.

Beneath the above, you are advised to have three folders:

* trunk
* tags
* branches

All current version support is development is done in trunk, development
on the next version is done in a separate branch in branches, and when a
release is made to customers, a "copy" is created in tags.

I use this setup myself, as follows:

c:\oracle\scripts\TOAD\\trunk
c:\oracle\scripts\TOAD\\tags
c:\oracle\scripts\TOAD\\branches

As you can see I have a separate repository for each project.

>> We have a production database and 4 development/test
>> databases that are cloned from production for various purposes.

So far, one repository is all you need. Each different
database/development team can check out the latest code from the
repository. The development teams might require a branch to do
development work though - to avoid messing up code already in
production, but then again, code already in production *should* have a
tag. Equally, it should be in trunk as well.

So your repository admin would create a copy of the current trunk and
tag it as "Release 6.66" (or whatever) and put it in the tags directory.
You now have a base copy of the 6.66 release that is going to
production. This can be used for bug fixes in this release but without
necessarily affecting development work on-going in trunk on the next
release.

>> We are a
>> private college and use the SCT Banner HE package. We maintain a few

>> modifications to the Banner delivered code, and also a number of
custom written add
>> ons--so we have package, view, table, etc, source that we
>> wish to maintain.
>>
>> o Do we set up one repository for our source, a production and test
>> repository, or a repository for each database instance?

One repository, I think. Your production code would be checked in
(imported) and immediately, a tag created in the tags directory as your
current base release of the production code.

Your production support team would work to fix bugs in this code,
probably in a copy of the base release in branches. All work they do to
fix bugs etc would be committed back to the branches part of the
repository.

Any development work would take place either in trunk (ie, starting from
the base release and going forward) or in a separate branch - assuming a
whole new start, or similar, exercise.

>> o What kind of directory structure works best for PL/SQL
>> development?...separate directories for different
>> schemas?...different object types?

I do all mine in one directory structure, I rarely have separate schemes
but if I do, I'd set up as follows (assumes trunk = development area):

c:\oracle\scripts\TOAD\\trunk\schema_a
c:\oracle\scripts\TOAD\\trunk\schema_b

All my work would take place there. When checking out my working copy,
I'd simply check out trunk - unless I'm only working on one particular
schema - do my work, make commits etc.

When it's ready for testing teams to take over, I create a tagged
"pre-release" in the tags directory which the test team export/check out
and play with. Any faults or bugs are fixed in trunk as normal and then
applied to the tagged pre-release.

Eventually, the release is passed as suitable for production and I
create a new Release tag and from that, create a release kit (if
necessary) for the production team (which is me!) to apply.

>> o What happens when we clone our production database to our
>> develop/test instances if we use Toads Team Coding?

I don't use Toad's Team Coding. So I don't have this problem. Under Toad
10.5 I use the Tortoise utility as my Third Party File Based Source
Control Provider. I also tend to use all the Tortoise commands *outside*
of Toad. Simply because I edit separate files in Toad to develop or bug
fix and I like to make sure that they all get committed as part of the
same (Subversion) transaction - rather than checking in one file at a
time from each tab.

I notice at 10.6, that I can also use the Subversion command line
utilities as a team coding provider - however, it appears that the
default settings for the subversion options are to "use locking on check
in/checkout" - this is against how Subversion works and is best avoided.

There is no need to lock a file on checkout and unlock it on check in -
so make sure the option is off.

>> o Will Team Coding work for our SQL Nav users?...when I try to check
>> out a file in SQL Nav it tells me the version of Team Coding
>> is too old even though I just installed it using the servers side
>> objects wizard in Toad 10.6.

Sorry, I don't use SQL Nav, however, if it too uses Subversion as the
source control utility, then it should be able to work happily alongside
Toad's own use of the SVN command line and, if you have it, other
people's use of TortoiseSVN and other SVN tools.

I rather suspect you need to bypass the Toad Team Coding stuff on the
database. With Subversion, I really don't think it's needed, and I've
used Subversion with Toad for a long time, and I've never created any
database objects with the Toad server side wizard - I'm not allowed to
for a start! Only application tables and objects are allowed on our
databases.

HTH

Cheers,
Norm. [TeamT]

Information in this message may be confidential and may be legally
privileged. If you have received this message by mistake, please notify the
sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should
still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to
under the Freedom of Information Act, Data Protection Act or for litigation.
Email messages and attachments sent to or from any Environment Agency
address may also be accessed by someone other than the sender or recipient,
for business purposes.

If we have sent you information and you wish to use it please read our terms
and conditions which you can get by calling us on 08708 506 506. Find out
more about the Environment Agency at www.environment-agency.gov.uk

#4

Hi Wendy,

Hi Norm, thank you for such a clear response–this is very
helpful. Wendy
Welcome.

I’m not sure if you are aware but CollabNet now have a system (for
Windows 32 and also 64 bit) called Subversion Edge. This gives you all
you need to run a subversion server on a windows box, with WebDAV
access, an Apache server etc etc. There’s a pretty neat admin tool that
lets you create and maintain repositories etc from your browser.

One thing though, it creates two services in Windows and you must change
these to run as “a dedicated logion” and not as local system. If you run
as local system the server can’t startup.

It’s in the readme but who reads those? :wink:

Cheers,
Norm. [TeamT]

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk


#5

Hi;

I try to import of the data into a Oracle 10g database in using the following
command :

imp SYSTEM /laposte+2 file=/oradata/ora48001/donnees/archive / hp_ntced_aa.dmp
log=/oradata/ora48001/donnees/archive / hp_ntced_aa.log fromuser=hp_ntced_aa
touser=HP_NTCED_AA commit=y ignore=y

But I have of error messages on constraints and indexes !

How realize the import procedure without know the schema of the database ?

I would like just to import only the data from a dump file.

Regards


#6

Morning djamel,

imp SYSTEM/laposte+2
file=/oradata/ora48001/donnees/archive/hp_ntced_aa.dmp
log=/oradata/ora48001/donnees/archive/hp_ntced_aa.log
fromuser=hp_ntced_aa touser=HP_NTCED_AA commit=y ignore=y

**
But I have of error messages on constraints and indexes !
*How realize the import procedure without know the schema of the
database ? *
I would like just to import only the data from a dump file.

add the following to your imp command:

indexes=no constraints=no

you may also decrease the import time by adding:

buffer=40960000

Adjust the value to suit your system, but the above works for me.

And, one last thing, please don’t hijack an existing thread to post a
question. If you do, people like me with threaded email clients may not
see it as it appears under a thread that we may not be watching. For a
new query, alsways start a new thread please. Thanks.

Cheers,
Norm. [TeamT]

Cheers,
Norm. [TeamT]


#7

Hi Norm,

Despite reading the documentation a number of times, I still don’t feel I have a
handle on how to set this up to work in our environment. None of the few
examples that I can find seem to meet the need…but I’m also working uphill in
that I don’t have a good feel for all the terminology involved, and it’s a
different paradigm from what we have used in the past (don’t suppose you ever
worked in a VMS environment using CMS?) So could you check my logic on this?

We use a packaged product, Banner, to run our Student Information Systems…it is
a huge software package with thousands of tables and objects. We customize some
of these, and in addition have a number of custom written add ons. We have a
production database and then 4 test instances that are cloned from the
production database. Typically we have one instance where we are testing and/or
customizing the next release of Banner from the vendor; an instance where we are
doing new development, and one where we might have production bugs being fixed.
All of the instances use all the same packages, tables, views, triggers…etc.
There are usually multiple programmers working on each instance, and each
programmer usually has more than one project they are working on within the
instance. Some objects are common to all the projects, like the student master
table, and common code libraries. We need to keep versions of source for the
production instance and each test instance and then be able to merge the test
changes back to production. We need to be able to move individual objects to
production as projects and maintenance are completed.

After reading the Subversion documentation, I’m thinking–in theory, that the
production source should be the Trunk and each test instance should be a Branch.
So then several programmers are working on a Branch and 1 finishes changes to a
package, can that one package be checked back into the branch and then be
committed back to the Trunk? Does a programmer check out individual objects to
work on them, or do they have the whole branch checked out, and just check in
the items they’ve altered?

When it’s ready for testing teams to take over, I create a tagged
“pre-release” in the tags directory which the test team export/check out

    and play with. Any faults or bugs are fixed in trunk as normal and then
    applied to the tagged pre-release.

So if one of our programmers completes a project in one of the test instances,
the changes could be could be saved to a ‘Tag’ without saving any of the changes
other programmers are in the process of making on that Branch?

Eventually, the release is passed as suitable for production and I
create a new Release tag and from that, create a release kit (if
necessary) for the production team (which is me!) to apply.

What is a release kit? Right now there is myself and one other programmer who
move changes to production. But part of the scope of this project is to
semi-automate the process and make the DBA’s responsible for moves to production
(and get the auditors off our backs). What the DBAs would like, is that checking
the source back into prod, would automagically trigger the update. I’m having a
hard time picturing this working smoothly.

On 10/22/2010 5:27 AM, Dunbar, Norman (Capgemini) wrote:

Hi Wendy,

>> Hi, I'm evaluating using Subversion with Toad and I'm trying
>> to figure out a reasonable fashion to configure the repository,
keeping it as
>> simple as possible, and I'm looking for suggestions others might
have?

Keep it simple! Always the best advice.

If you read Subversion manuals, or look at many repositories, there is a
"standard" setup whereby you can either:

* have one single repository with all your projects in; or
* have one repository per project.

Beneath the above, you are advised to have three folders:

* trunk
* tags
* branches

All current version support is development is done in trunk, development
on the next version is done in a separate branch in branches, and when a
release is made to customers, a "copy" is created in tags.

I use this setup myself, as follows:

c:\oracle\scripts\TOAD\\trunk
c:\oracle\scripts\TOAD\\tags
c:\oracle\scripts\TOAD\\branches

As you can see I have a separate repository for each project.

>> We have a production database and 4 development/test
>> databases that are cloned from production for various purposes.

So far, one repository is all you need. Each different
database/development team can check out the latest code from the
repository. The development teams might require a branch to do
development work though - to avoid messing up code already in
production, but then again, code already in production *should* have a
tag. Equally, it should be in trunk as well.

So your repository admin would create a copy of the current trunk and
tag it as "Release 6.66" (or whatever) and put it in the tags directory.
You now have a base copy of the 6.66 release that is going to
production. This can be used for bug fixes in this release but without
necessarily affecting development work on-going in trunk on the next
release.

>> We are a
>> private college and use the SCT Banner HE package. We maintain a few

>> modifications to the Banner delivered code, and also a number of
custom written add
>> ons--so we have package, view, table, etc, source that we
>> wish to maintain.
>>
>> o Do we set up one repository for our source, a production and test
>> repository, or a repository for each database instance?

One repository, I think. Your production code would be checked in
(imported) and immediately, a tag created in the tags directory as your
current base release of the production code.

Your production support team would work to fix bugs in this code,
probably in a copy of the base release in branches. All work they do to
fix bugs etc would be committed back to the branches part of the
repository.

Any development work would take place either in trunk (ie, starting from
the base release and going forward) or in a separate branch - assuming a
whole new start, or similar, exercise.

>> o What kind of directory structure works best for PL/SQL
>> development?...separate directories for different
>> schemas?...different object types?

I do all mine in one directory structure, I rarely have separate schemes
but if I do, I'd set up as follows (assumes trunk = development area):

c:\oracle\scripts\TOAD\\trunk\schema_a
c:\oracle\scripts\TOAD\\trunk\schema_b

All my work would take place there. When checking out my working copy,
I'd simply check out trunk - unless I'm only working on one particular
schema - do my work, make commits etc.

When it's ready for testing teams to take over, I create a tagged
"pre-release" in the tags directory which the test team export/check out
and play with. Any faults or bugs are fixed in trunk as normal and then
applied to the tagged pre-release.

Eventually, the release is passed as suitable for production and I
create a new Release tag and from that, create a release kit (if
necessary) for the production team (which is me!) to apply.

>> o What happens when we clone our production database to our
>> develop/test instances if we use Toads Team Coding?

I don't use Toad's Team Coding. So I don't have this problem. Under Toad
10.5 I use the Tortoise utility as my Third Party File Based Source
Control Provider. I also tend to use all the Tortoise commands *outside*
of Toad. Simply because I edit separate files in Toad to develop or bug
fix and I like to make sure that they all get committed as part of the
same (Subversion) transaction - rather than checking in one file at a
time from each tab.

I notice at 10.6, that I can also use the Subversion command line
utilities as a team coding provider - however, it appears that the
default settings for the subversion options are to "use locking on check
in/checkout" - this is against how Subversion works and is best avoided.

There is no need to lock a file on checkout and unlock it on check in -
so make sure the option is off.

>> o Will Team Coding work for our SQL Nav users?...when I try to check
>> out a file in SQL Nav it tells me the version of Team Coding
>> is too old even though I just installed it using the servers side
>> objects wizard in Toad 10.6.

Sorry, I don't use SQL Nav, however, if it too uses Subversion as the
source control utility, then it should be able to work happily alongside
Toad's own use of the SVN command line and, if you have it, other
people's use of TortoiseSVN and other SVN tools.

I rather suspect you need to bypass the Toad Team Coding stuff on the
database. With Subversion, I really don't think it's needed, and I've
used Subversion with Toad for a long time, and I've never created any
database objects with the Toad server side wizard - I'm not allowed to
for a start! Only application tables and objects are allowed on our
databases.

HTH

Cheers,
Norm. [TeamT]

Information in this message may be confidential and may be legally
privileged. If you have received this message by mistake, please notify the
sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should
still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to
under the Freedom of Information Act, Data Protection Act or for litigation.
Email messages and attachments sent to or from any Environment Agency
address may also be accessed by someone other than the sender or recipient,
for business purposes.

If we have sent you information and you wish to use it please read our terms
and conditions which you can get by calling us on 08708 506 506. Find out
more about the Environment Agency at www.environment-agency.gov.uk

#8

Evening Wendy,

you have just caught me as I’m about to leave for the evening, so excuse
the brevity of my reply. I’ll have a better think overnight and
hopefully get a better reply to you tomorrow.

Despite reading the documentation a number of times,
Have you downloaded the various books available on Subversion. There’s
the official Subversion book http://svnbook.red-bean.com/ and the Brice
Perens Open Source Series:
http://www.phptr.com/content/images/0131855182/downloads/Nagel_book.pdf

  • these are very useful.

There is a good one from APRESS called Practical Subversion which is
also good. I think there’s a PDF version as well as “dead tree”.

I still don’t feel
I have a handle on how to set this up to work in our environment. None
of the few examples that I can find seem to meet the need…but I’m
also working uphill in that I don’t have a good feel for all the
terminology involved, and it’s a different paradigm from what we have
used in the past (don’t suppose you ever worked in a VMS environment
using CMS?) So could you check my logic on this?
I know how you feel. It’s new, strange and not what you are used to. And
no, I haven’t worked on VMS since around 1982 at college and I loath VMS
with a vengeance!

We use a packaged product, Banner, to run our Student Information
Systems…it is a huge software package with thousands of tables and
objects.
have a look at the “Third Party” stuff in the book or books above. There
is a chapter (at least) on how to handle situations like this.

We customize some of these, and in addition have a number of
custom written add ons. We have a production database and then 4 test
instances that are cloned from the production database. Typically we
have one instance where we are testing and/or customizing the next
release of Banner from the vendor; an instance where we are doing new
development, and one where we might have production bugs being fixed.
All of the instances use all the same packages, tables, views,
triggers…etc. There are usually multiple programmers working on each
instance, and each programmer usually has more than one project they are
working on within the instance. Some objects are common to all the
projects, like the student master table, and common code libraries. We
need to keep versions of source for the production instance and each
test instance and then be able to merge the test changes back to
production. We need to be able to move individual objects to production
as projects and maintenance are completed.
I’ll have a think about this.

After reading the Subversion documentation, I’m thinking–in theory,
that the production source should be the Trunk and each test instance
should be a Branch. So then several programmers are working on a Branch
and 1 finishes changes to a package, can that one package be checked
back into the branch and then be committed back to the Trunk?
There is no check-in and commit, you simply commit. But yes, all that
developer 1 does is :

  • finish testing and editing and testing etc.
  • svn update
  • svn commit -m “Commit message”

Now, the update gets the developer the latest snapshot of the repository
and brings down all other developers committed changes. There may be none.

The commit applies his/her own changes. And, provided that there are no
conflict to be resolved, puts everything into the repository.

You are permitted to have as many people as you like working on the same
file/files and as long as the changes by each don’t overlap, Subversion
will happily merge your changes with mine and with anyone else’s when
either an update or commit is executed.

Does a
programmer check out individual objects to work on them, or do they have
the whole branch checked out, and just check in the items they’ve altered?
Either or. You choose.
You can checkout the entire repository, a folder within it, a branch or
tag (or parts thereof) a single file. Anything you need.

/When it’s ready for testing teams to take over, I create a tagged/
/“pre-release” in the tags directory which the test team
export/check out/
/and play with. Any faults or bugs are fixed in trunk as normal
and then/
/applied to the tagged pre-release./

So if one of our programmers completes a project in one of the test
instances, the changes could be could be saved to a ‘Tag’ without saving
any of the changes other programmers are in the process of making on
that Branch?
Yes, but they must be committed back into the repository first, then a
tag (or branch) created.

/Eventually, the release is passed as suitable for
production and I/
/create a new Release tag and from that, create a release
kit (if/
/necessary) for the production team (which is me!) to apply./

What is a release kit?
All the files, documents and whatever required to apply the changes to
the test or production system.

Right now there is myself and one other
programmer who move changes to production. But part of the scope of
this project is to semi-automate the process and make the DBA’s
responsible for moves to production (and get the auditors off our
backs). What the DBAs would like, is that checking the source back
into prod, would automagically trigger the update. I’m having a hard
time picturing this working smoothly.
You need something like a hook script running on the Subversion server.
When something is committed, the hook can run a check-out and build to
prove that the committed code doesn’t “break the build” etc. If you use
a precommit hook, you can set a status to prevent the commit if it
detects problems. etc etc.

I’ll see what I can dig up for you when I understand how your
development system works alongside the third party stuff. Etc.

Cheers,
Norm. [TeamT]


#9

Morning Wendy,

A pretty decent book I never mentioned yesterday is Pragmatic Version
Control With Subversion. It has a chapter on how to integrate third
party code into your repository. Looks pretty useful to me. A couple of
drawbacks though - around what to do when the vendor releases a new
release. However…

We use a packaged product, Banner, to run our Student
Information Systems…it is a huge software package with
thousands of tables and objects. We customize some of
these, and in addition have a number of custom written add
ons. We have a production database and then 4 test
instances that are cloned from the production database.
Typically we have one instance where we are testing and/or
customizing the next release of Banner from the vendor; an
instance where we are doing new development, and one where
we might have production bugs being fixed. All of the
instances use all the same packages, tables, views,
triggers…etc. There are usually multiple programmers
working on each instance, and each programmer usually has
more than one project they are working on within the
instance. Some objects are common to all the projects, like
the student master table, and common code libraries.

Here are my assumptions/observations from the above:

  • You get regular releases of BANNER from the vendor.

  • You customise the code by changing files within the released code,
    adding files, and possibly deleting files.

  • At any time after this release, your well oiled teams spring into
    action and you have the following work progressing all at the same time:

    • Production - Code is never changed except when fully tested and QA’d
      releases arrive via the TEST 4 database systems below.

    • Test 1 - testing the latest BANNER release from the vendor.

    • Test 2 - New development work.

    • Test 3 - Customising the latest BANNER release.

    • Test 4 - Fixing production bugs. (Again?)

  • Teams of multiple developers are working on one or more of the above.

  • Multiple “projects” exists within each development area.

  • You have a lot of common code spread over all these databases.

Ok, dealing with production first. Easy. The code in production is a mix
of the BANNER supplied code, your own deletions, amendments and
additions and should have come through only from the Test 4 system.
Your production code is basically what you will have in trunk.

You need to make sure that no-one trashes trunk and that any commits to
trunk don’t break the compile. If they do, the developer responsible
should be “publicly humiliated” :wink:

Test 1 is another easy one. The code drop from the vendor should be
extracted from it’s various tar/zip files into a new folder. Once done,
that folder can be imported into the repository as a tag - something
like “Banner release a.b.c.d”.

It would be nice if a new release of the vendor code could be extracted
over the top of the last release. This would let Subversion know what
has changed - unfortunately, deleted files would still remain from the
previous release. You CANNOT simply delete a folder and recreate it and
add it back in again. Subversion is not happy when you do that.

“Pragmatic” gives details of a way in which this can be done, but it
relies on a Perl script. I have not looked into this though.

What you could do is have a separate repository for the vendor code
drops. All you do is extract, import into a tag and that’ sit. You would
obviously publish the URL of the appropriate tags to the developers so
that they can recreate their development set up in the Test 1 and Test 3
areas.

Test 2 is the biggie, you have all your development work going on here
and various projects within this environment. I assume that the files in
the BANNER release are the ones being used by all projects.

Test 3 looks interesting. I’d set up the system so that the
administrators check out (or export) the latest BANNER release from the
vendor repository and then import it into their own development
repository. This would most likely be on a branch off of trunk (Trunk is
production level code!)

Test 4 is a simple one. Create a branch of trunk and have the developers
work on it by checking out form the branch not from trunk. No-one should
check out the trunk really except maybe the people creating a full code
release (BANNER, your stuff etc) ready for application to production (as
a full release).

There is an alternative way, getting Subversion to create patches to
bring production (trunk) up to a Test 4 branch. Then simply apply the
patches to the running code in production. Either way works.

A repository setup to cover the above might be, and assuming a separate
repository for the BANNER drops:

repos/vendor/banner/releases

Within this repository, you would have tags for each release in a
standard format:

repos/vendor/banner/releases/1.0.0.0
repos/vendor/banner/releases/1.0.1.0
repos/vendor/banner/releases/2.0.0.0

There is no trunk, tags or branches here, simply releases. No work is
ever done here, this repository exists purely to keep safe the code as
originally supplied by the vendors.

Next, a development repository with the “standard” trunk/tags/branches
folders.

repos/banner/trunk
repos/banner/branches
repos/banner/tags

As mentioned above, production code is in trunk and is untouchable!
Branches is where you would create the development stuff by copying
trunk into a development branch with an appropriate name. Your
developers would checkout and work with this branch’s code.

Any time a fix is applied to production (in trunk) you can easily merge
that change back into the development branch - if appropriate.

You need, with the above setup, to make sure that the development branch
has a copy of the vendor code appropriate within it, especially if you
are making changes - the last thing you want a developer to do is change
code in the vendor drop repository! The administrator should load up
(import) the vendor drop into the development branch before letting the
hoards of great unwashed developers (!) get access to it. (I consider
myself one of the great unwashed!)

So, how does the above work? Starting from a new code release from the
vendor:

  1. Unpack the release into a new folder.

mkdir banner_1.2.3.4
cd banner_1.2.3.4
untar …/banner.1.2.3.4_release.tar

  1. Import into a new “release” in the repos/vendor/banner/releases
    repository.

svn import . svn://repos/vendor/banner/releases/1.2.3.4

  1. Import the same code into a new branch in the main development
    repository. This is for the Test 3 area.

svn import . svn://repos/banner/branches/1.2.3.4

  1. At this point, you have a safe copy in one repository and a “soon to
    be butchered” copy in a new branch of the main repository. No-one knows
    about it yet so can’t access it. Until you tell them!

  2. Advise the Test 3 team that there is a new code drop available for
    them to play with, tell them the URL is
    svn://repos/banner/branches/1.2.3.4.

They can no check out the new code, as supplied, and begin to apply
changes. As the commit changes, it goes back into the branch just
created for this very purpose. The previous branch still exists and is
still available for bug fixes.

Equally, fixes or changes applied to the previous release can be merged
into this branch - in the event that the same changes are made each time
a new release is made.

  1. The Test 2 team are eager to get on with their own butchering of the
    new release, so when Test 3 are done playing (!), the repository admin
    can create a new branch based on the Test 3 team’s work so far. This is
    a simple enough process:

svn copy svn://repos/banner/branches/1.2.3.4
svn://repos/banner/branches/1.2.3.4_development

Now this is a lazy copy in as much as nothing is actually copied within
the repository, just a note is made that the copy has been made. The
correct files will be checked out when a developer checks out the new
branch.

  1. Test 2 developers now check out the new development branch:

c:
cd development
mkdir 1.2.3.4
cd 1.2.3.4
svn checkout svn://repos/banner/branches/1.2.3.4_develoment ./

Now all the work that goes on here is committed back into the
development branch and doesn’t affect the work done (and possibly
continuing to be done) by Test 3. Any fixes made by Test 3 can be easily
merged into the Test 2 branch as and when required. And indeed, Test 2
work can be merged into Test 3 work too.

I think that will cover it. You may see something I don’t, give me a
shout if so.

Ok, all development work is done, and you need to update production.
Easy! You can check the logs for all changes made since the code was
branches and apply those in bulk or in batches as required using the
merge command.

A quick compile and test by test scripts etc, the QA team whatever, and
it’s ready to go.

You need a full release of the code now in trunk for a new
customer/database setup, easy, export it:

c:
cd production
mkdir 1.2.3.4
cd 1.2.3.4
svn export svn://repos/banner/trunk ./

You now have all the files in the latest production release with the
original BANNER files modified, added to, deleted or whatever, plus all
changes made by your Test 3 team, the TEST 2 team and so on. All tested
and approved and ready to be run against the database. (Ok, you might
have problems at this point because if the release consists of CREATE
TABLE scripts then you may have those tables already existing!)

If all you want is a patch kit to update a running production system to
the latest 1.2.3.4 release then you would use the svn diff command to
list differences between the current HEAD revision and the original
branch.

I’m not giving an example here because (a) I’ve never used it and (b) I
can’t test it at the moment! Plus I rather suspect this won’t be an
option for you.

We need to keep versions of source for the production instance
and each test instance and then be able to merge the test
changes back to production.
Separate branches will facilitate this requirement.

We need to be able to move
individual objects to production as projects and maintenance
are completed.
Easy to do. Simply export when ready and you have a release kit. You can
export an entire project or folder or file within it as you need.

HTH

Cheers,
Norm. [TeamT]

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk


#10

Hi Wendy,

oops! I missed out Test 4 - production bug fixers.

When a bug is found in production, it must be fixed. You can branch off
a new branch and copy the files from trunk - either all of it, or some
of it.

The Test 4 teams would work there to fix the bug, in the branch, and
commit changes back to the branch.

Once fixed and tested, the changes could be merged into the trunk and
the working branch deleted as it is no longer required.

Cheers,
Norm. [TeamT]


#11

Wow, someone else with our same problem! We recently (earlier in 2010) converted
from VMS as a batch job platform using CMS code control and JSS job scheduling,
to Linux, SubVersion, and Autosys.

Norm has excellent advice, of course. For production deployment, our conversion
team developed some scripts that run on the Linux platform that make that part
(deployment) even easier. I have the whole SubVersion repository for the
application checked out to a directory structure on my PC. (Remember that Check
Out doesn’t do any sort of exclusive lock like a Reserve used to on CMS;
it just means I have a copy and SubVersion is keeping track of where in the
repository it came from.) When I need to change a PL/SQL program, for example, I
first do a SubVersion Update to make sure I have the latest version (such a
silly requirement; CMS always had the latest version, I didn’t have to
remember to ask for it! ), then simply load the file(s) in Toad,
connect to the development database, and edit/compile/test away until I like
what I see. I then (SVN update again, as Norm said, and then) SVN commit my
changes to the repository, using my change request number as the first word of
the SVN commit comment.

Then the magic takes over. I log on to Linux and execute the
“package” script, giving it my change request number. This script
runs SubVersion on Linux, finds all modules (files) that have the specified
number in their comments, pulls them into a temporary directory, and combines
them into a tar file (multi-file archive, for those who don’t know, kind
of like a zip file in Windows). Operations can then execute the
“promote” script, giving it the tar file filename, and this script
will copy the tar file to the next environment (Dev to Test, Test to Pre-Prod,
Pre-Prod to Prod), unpack all the files, put them in the right places (there can
be more than just PL/SQL files: Linux script files, SQL scripts, etc.), and for
PL/SQL modules it will log on to the database and compile the code, and then run
another script that looks for any invalid objects and compiles them. Pretty
slick.

Nate Schroeder

Enterprise Services - Data Management Team

Monsanto Company

800 N. Lindbergh Blvd. LC4D - Saint Louis, MO - 63167

314-694-2592


#12

Morning Nate,

Norm has excellent advice, of course.
Thanks.

… I first do a SubVersion Update to make
sure I have the latest version (such a silly requirement;
You don’t have to do an update before you commit, but if changes have
been made and committed by other developers you will get a message that
your copy of the file(s) being committed is/are out of date and that you
should do an update now.

CMS always had the latest version, I didn’t have to remember
to ask for it! ),
Interesting. I wonder if it simply does one in the background when you
commit something?

… connect to the development database, and
edit/compile/test away until I like what I see. I then (SVN
update again, as Norm said, and then) SVN commit my changes
to the repository, using my change request number as the
first word of the SVN commit comment.
I forgot to mention in my initial blurb, that you should always re-test
if you find that the update changed anything in your workspace.

Then the magic takes over. I log on to Linux and execute
the “package” script, giving it my change request number.
I wonder if you can use hook scripts to do this for you automagically?
In fact, I’m pretty certain that you can. I have seen an example where
the pre-commit hook won’t let a commit finish until/unless it passes a
full checkout and compile test.

I suspect you could use a post-commit hook to do a similar task to that
which you are doing now? That means even more magic!

Cheers,
Norm. [TeamT]

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk


#13

Learn more every day - thanks. I don’t know SubVersion hook scripts.
Does a post-commit hook script need to run on the same machine the
commit happens on? In this case I’m committing from a PC but the script
runs on the Linux box.

CMS always had the latest version, I didn’t have to remember
to ask for it! ),
Interesting. I wonder if it simply does one in the background when you
commit something?
Remember, in VMS you’re on a multiuser computer. I don’t have a private
copy of the whole repository; the repository is just there, in the
CMSREF directory. CMS keeps that repository up to date.

Nate Schroeder
Enterprise Services - Data Management Team
Monsanto Company
800 N. Lindbergh Blvd. LC4D - Saint Louis, MO - 63167
314-694-2592


#14

Hi Nate,

Learn more every day - thanks. I don’t know Subversion hook scripts.
It has lots of them!

Does a post-commit hook script need to run on the same machine the
commit happens on? In this case I’m committing from a PC but
the script runs on the Linux box.
The hook runs on the same box that the repository is on, so it would run
on your Linux box I expect.

Cheers,
Norm. [TeamT]

Information in this message may be confidential and may be legally privileged. If you have received this message by mistake, please notify the sender immediately, delete it and do not copy it to anyone else.

We have checked this email and its attachments for viruses. But you should still check any attachment before opening it.
We may have to make this message and any reply to it public if asked to under the Freedom of Information Act, Data Protection Act or for litigation. Email messages and attachments sent to or from any Environment Agency address may also be accessed by someone other than the sender or recipient, for business purposes.

If we have sent you information and you wish to use it please read our terms and conditions which you can get by calling us on 08708 506 506. Find out more about the Environment Agency at www.environment-agency.gov.uk


#15

Wow, that is pretty slick…and also pretty much exactly what we are trying to
do!

On 10/27/2010 3:23 PM, SCHROEDER, NATHAN E [AG/1000] wrote:

Wow, someone else with our same problem! We recently (earlier in 2010)
converted from VMS as a batch job platform using CMS code control and JSS
job scheduling, to Linux, SubVersion, and Autosys.

Norm has excellent advice, of course. For production deployment, our
conversion team developed some scripts that run on the Linux platform that
make that part (deployment) even easier. I have the whole SubVersion
repository for the application checked out to a directory structure on my
PC. (Remember that Check Out doesn’t do any sort of exclusive lock
like a Reserve used to on CMS; it just means I have a copy and SubVersion is
keeping track of where in the repository it came from.) When I need to
change a PL/SQL program, for example, I first do a SubVersion Update to make
sure I have the latest version (such a silly requirement; CMS always had the
latest version, I didn’t have to remember to ask for it!
), then simply load the file(s) in Toad, connect to the
development database, and edit/compile/test away until I like what I see. I
then (SVN update again, as Norm said, and then) SVN commit my changes to the
repository, using my change request number as the first word of the SVN
commit comment.

Then the magic takes over. I log on to Linux and execute the
“package” script, giving it my change request number. This
script runs SubVersion on Linux, finds all modules (files) that have the
specified number in their comments, pulls them into a temporary directory,
and combines them into a tar file (multi-file archive, for those who
don’t know, kind of like a zip file in Windows). Operations can then
execute the “promote” script, giving it the tar file filename,
and this script will copy the tar file to the next environment (Dev to Test,
Test to Pre-Prod, Pre-Prod to Prod), unpack all the files, put them in the
right places (there can be more than just PL/SQL files: Linux script files,
SQL scripts, etc.), and for PL/SQL modules it will log on to the database
and compile the code, and then run another script that looks for any invalid
objects and compiles them. Pretty slick.

Nate Schroeder

Enterprise Services - Data Management Team

Monsanto Company

800 N. Lindbergh Blvd. LC4D - Saint Louis, MO - 63167

314-694-2592